一些agent数据处理的问题，model_runtime模块的说明文档 (#3943)

* 一些agent数据出来的问题

* Changes:
- Translated and updated the Model Runtime documentation to reflect the latest changes and features.
- Clarified the decoupling benefits of the Model Runtime module from the Chatchat service.
- Removed outdated information regarding the model configuration storage module.
- Detailed the retained functionalities post-removal of the Dify configuration page.
- Provided a comprehensive overview of the Model Runtime's three-layered structure.
- Included the status of the `fetch-from-remote` feature and its non-implementation in Dify.
- Added instructions for custom service provider model capabilities.

chatchat-server/chatchat/server/agent/agent_factory/glm3_agent.py

@@ -95,7 +95,7 @@ def create_structured_glm3_chat_agent(
             "parameters": parameters
         }
         tools_json.append(simplified_config_langchain)
-    tools = "\n".join([str(tool) for tool in tools_json])
+    tools = "\n".join([json.dumps(tool, indent=4, ensure_ascii=False) for tool in tools_json])
 
     prompt = ChatPromptTemplate(
         input_variables=["input", "agent_scratchpad"],

chatchat-server/chatchat/server/chat/chat.py

@@ -7,7 +7,7 @@
 from fastapi import Body
 from sse_starlette.sse import EventSourceResponse
 from langchain_core.output_parsers import StrOutputParser
-from langchain_core.messages import AIMessage, HumanMessage
+from langchain_core.messages import AIMessage, HumanMessage, convert_to_messages
 
 from langchain.chains import LLMChain
 from langchain.prompts.chat import ChatPromptTemplate
@@ -136,11 +136,17 @@ async def chat_iterator() -> AsyncIterable[OpenAIChatOutput]:
                                           history=history,
                                           history_len=history_len,
                                           metadata=metadata)
+
+        _history = [History.from_data(h) for h in history]
+        chat_history = [h.to_msg_tuple() for h in _history]
+
+        history_message = convert_to_messages(chat_history)
+
         task = asyncio.create_task(wrap_done(
             full_chain.ainvoke(
                 {
                     "input": query,
-                    "chat_history": [],
+                    "chat_history": history_message,
                 }
             ), callback.done))
 

model-providers/model_providers/core/model_runtime/README.md

@@ -1,9 +1,13 @@
 # Model Runtime
 
-This module provides the interface for invoking and authenticating various models, and offers Dify a unified information and credentials form rule for model providers.
+This module is the loading module that unifies Dify's supplier information and credential form. The project has an OpenAI service forwarding capability at the upper layer, which can support the conversion of supplier services to OpenAI EndPoint calls.
 
-- On one hand, it decouples models from upstream and downstream processes, facilitating horizontal expansion for developers,
-- On the other hand, it allows for direct display of providers and models in the frontend interface by simply defining them in the backend, eliminating the need to modify frontend logic.
+On the basis of the original, this module has added `VIewProfileConfig`, `RESTFulProfileServer`, `OpenAIPluginsClient`, and has rewritten the logic for loading supplier configuration, focusing only on the supplier's configuration. The model configuration and calling logic are implemented in this module, providing a separation of model calls for `Chatchat`. The benefits of doing so are:
+- It changes the decoupling method between the model and the upstream and downstream callers.
+- During development, one does not need to pay attention to the specific implementation of the service provider module, but can simply make calls through the OpenAI EndPoint provided by Model Runtime.
+
+> Please note! Because the model configuration storage module has been removed, the ability to operate suppliers and models on the front-end page has been deleted accordingly. Now, it is only possible to configure suppliers and models through a backend yaml file.
+> Only a supplier configuration details interface is provided, and in the future, we will consider providing a model configuration details interface through other means.
 
 ## Features
 
@@ -18,53 +22,41 @@ This module provides the interface for invoking and authenticating various model
 
 - Model provider display
 
-  ![image-20231210143654461](./docs/en_US/images/index/image-20231210143654461.png)
-
   Displays a list of all supported providers, including provider names, icons, supported model types list, predefined model list, configuration method, and credentials form rules, etc. For detailed rule design, see: [Schema](./schema.md).
 
-- Selectable model list display
-
-  ![image-20231210144229650](./docs/en_US/images/index/image-20231210144229650.png)
-
-  After configuring provider/model credentials, the dropdown (application orchestration interface/default model) allows viewing of the available LLM list. Greyed out items represent predefined model lists from providers without configured credentials, facilitating user review of supported models.
-
-  In addition, this list also returns configurable parameter information and rules for LLM, as shown below:
-
-  ![image-20231210144814617](./docs/en_US/images/index/image-20231210144814617.png)	
-
-  These parameters are all defined in the backend, allowing different settings for various parameters supported by different models, as detailed in: [Schema](./docs/en_US/schema.md#ParameterRule).
-
 - Provider/model credential authentication
-
-  ![image-20231210151548521](./docs/en_US/images/index/image-20231210151548521.png)
-
-  ![image-20231210151628992](./docs/en_US/images/index/image-20231210151628992.png)
-
-  The provider list returns configuration information for the credentials form, which can be authenticated through Runtime's interface. The first image above is a provider credential DEMO, and the second is a model credential DEMO.
+  The provider list returns configuration information for the credentials form, which can be authenticated through Runtime's interface. 
 
 ## Structure
 
-![](./docs/en_US/images/index/image-20231210165243632.png)
+![](./docs/en_US/images/index/img.png)
 
 Model Runtime is divided into three layers:
 
-- The outermost layer is the factory method
+- The outermost layer is the OpenAI EndPoint publishing layer
 
-  It provides methods for obtaining all providers, all model lists, getting provider instances, and authenticating provider/model credentials.
+  It provides asynchronous loading configuration `VIewProfileConfig`
+  Supplier service publishing `RESTFulProfileServer`
 
 - The second layer is the provider layer
 
   It provides the current provider's model list, model instance obtaining, provider credential authentication, and provider configuration rule information, **allowing horizontal expansion** to support different providers.
+  For supplier/model credentials, there are two situations:
+  - For centralized suppliers like OpenAI, you need to define authentication credentials like **api_key**.
+  - For locally deployed suppliers like [**Xinference**](https://github.com/xorbitsai/inference), you need to define address credentials like **server_url**. Sometimes you also need to define model type credentials like **model_uid**.
 
 - The bottom layer is the model layer
 
-  It offers direct invocation of various model types, predefined model configuration information, getting predefined/remote model lists, model credential authentication methods. Different models provide additional special methods, like LLM's pre-computed tokens method, cost information obtaining method, etc., **allowing horizontal expansion** for different models under the same provider (within supported model types).
+  It provides direct calls for various model types, predefined model configuration information, obtaining predefined/remote model lists, model credential authentication methods, and different models provide special methods, such as LLM providing pre-computed tokens methods, obtaining cost information methods, etc., **which can be scaled horizontally** for different models under the same supplier (under the supported model types).
+
+  Here we need to distinguish between model parameters and model credentials first.
 
+  - Model parameters (**defined in this layer**): These are parameters that often need to be changed and adjusted at any time, such as LLM's **max_tokens**, **temperature**, etc. These parameters are adjusted by users on the front-end page, so it is necessary to define the rules of parameters on the backend to facilitate the display and adjustment on the front-end page. In DifyRuntime, their parameter names are generally **model_parameters: Dict[str, any]**.
 
+  - Model credentials (**defined in the supplier layer**): These are parameters that do not often change and generally do not change after configuration, such as **api_key**, **server_url**, etc. In DifyRuntime, their parameter names are generally **credentials: Dict[str, any]**. The credentials of the Provider layer will be directly passed to this layer, and there is no need to define them separately.
 
-## Next Steps
+ 
 
-- Add new provider configuration: [Link](./docs/en_US/provider_scale_out.md)
-- Add new models for existing providers: [Link](./docs/en_US/provider_scale_out.md#AddModel)
-- View YAML configuration rules: [Link](./docs/en_US/schema.md)
-- Implement interface methods: [Link](./docs/en_US/interfaces.md)
+### If you want to implement a custom service provider model capability
+- [Go here 👈🏻](./docs/en_US/interfaces.md)
+- [Details about custom models](./docs/en_US/provider_scale_out.md)

model-providers/model_providers/core/model_runtime/README_CN.md

@@ -1,9 +1,15 @@
+
 # Model Runtime
 
-该模块提供了各模型的调用、鉴权接口，并为 Dify 提供了统一的模型供应商的信息和凭据表单规则。
+该模块是Dify 统一供应商信息和凭据表单的加载模块，此项目在上层做了openai服务转发能力，可以支持对供应商服务转换为 OpenAI EndPoint的调用
+
+该模块在原有的基础上，增加了`VIewProfileConfig`、`RESTFulProfileServer`、`OpenAIPluginsClient`重写了供应商配置加载逻辑，仅关注供应商的配置，
+模型的配置和调用逻辑在本模块实现， 为`Chatchat`提供了模型调用的分离，这样做的好处是：
+- 改变了模型和上下游调用方解耦方式
+- 在开发时，不需要关注服务商模块的具体实现，仅通过Model Runtime提供的OpenAI EndPoint调用即可
 
-- 一方面将模型和上下游解耦，方便开发者对模型横向扩展，
-- 另一方面提供了只需在后端定义供应商和模型，即可在前端页面直接展示，无需修改前端逻辑。
+> 请注意！因为删除了模型配置的存储模块，原本在前端页面操作供应商和模型的能力因此被删除，现在只能通过后端yaml文件配置供应商和模型，
+> 仅提供了供应商的配置详情接口，未来会考虑通过其他方式提供模型的配置详情接口
 
 ## 功能介绍
 
@@ -16,53 +22,32 @@
   - `Text-to-speech Model` - 文本转语音能力
   - `Moderation` - Moderation 能力
 
+### 删除Dify配置页面的情况下，保留了以下功能：
 - 模型供应商展示
+展示所有已支持的供应商列表，除了返回供应商名称、图标之外，还提供了支持的模型类型列表，预定义模型列表、配置方式以及配置凭据的表单规则等等，规则设计详见：[Schema](./docs/zh_Hans/schema.md)。
 
-  ![image-20231210143654461](./docs/zh_Hans/images/index/image-20231210143654461.png)
-
-​	展示所有已支持的供应商列表，除了返回供应商名称、图标之外，还提供了支持的模型类型列表，预定义模型列表、配置方式以及配置凭据的表单规则等等，规则设计详见：[Schema](./docs/zh_Hans/schema.md)。
-
-- 可选择的模型列表展示
-
-  ![image-20231210144229650](./docs/zh_Hans/images/index/image-20231210144229650.png)
-
-​	配置供应商/模型凭据后，可在此下拉（应用编排界面/默认模型）查看可用的 LLM 列表，其中灰色的为未配置凭据供应商的预定义模型列表，方便用户查看已支持的模型。
-
-​	除此之外，该列表还返回了 LLM 可配置的参数信息和规则，如下图：
-
-​	![image-20231210144814617](./docs/zh_Hans/images/index/image-20231210144814617.png)	
-
-​	这里的参数均为后端定义，相比之前只有 5 种固定参数，这里可为不同模型设置所支持的各种参数，详见：[Schema](./docs/zh_Hans/schema.md#ParameterRule)。
-
-- 供应商/模型凭据鉴权
-
-  ![image-20231210151548521](./docs/zh_Hans/images/index/image-20231210151548521.png)
-
-![image-20231210151628992](./docs/zh_Hans/images/index/image-20231210151628992.png)
-
-​	供应商列表返回了凭据表单的配置信息，可通过 Runtime 提供的接口对凭据进行鉴权，上图 1 为供应商凭据 DEMO，上图 2 为模型凭据 DEMO。
+- 供应商/模型凭据鉴权 
+ 供应商列表返回了凭据表单的配置信息，可通过 Runtime 提供的接口对凭据进行鉴权， 
 
 ## 结构
 
-![](./docs/zh_Hans/images/index/image-20231210165243632.png)
+![](./docs/zh_Hans/images/index/img.png)
 
 Model Runtime 分三层：
 
-- 最外层为工厂方法
+- 最外层为OpenAI EndPoint发布层
 
-  提供获取所有供应商、所有模型列表、获取供应商实例、供应商/模型凭据鉴权方法。
+  提供异步加载配置`VIewProfileConfig`
+  供应商服务发布`RESTFulProfileServer` 
 
 - 第二层为供应商层
 
   提供获取当前供应商模型列表、获取模型实例、供应商凭据鉴权、供应商配置规则信息，**可横向扩展**以支持不同的供应商。
 
   对于供应商/模型凭据，有两种情况
   - 如OpenAI这类中心化供应商，需要定义如**api_key**这类的鉴权凭据
-  - 如[**Xinference**](https://github.com/xorbitsai/inference)这类本地部署的供应商，需要定义如**server_url**这类的地址凭据，有时候还需要定义**model_uid**之类的模型类型凭据，就像下面这样，当在供应商层定义了这些凭据后，就可以在前端页面上直接展示，无需修改前端逻辑。
-  ![Alt text](docs/zh_Hans/images/index/image.png)
-
-  当配置好凭据后，就可以通过DifyRuntime的外部接口直接获取到对应供应商所需要的**Schema**（凭据表单规则），从而在可以在不修改前端逻辑的情况下，提供新的供应商/模型的支持。
-
+  - 如[**Xinference**](https://github.com/xorbitsai/inference)这类本地部署的供应商，需要定义如**server_url**这类的地址凭据，有时候还需要定义**model_uid**之类的模型类型凭据， 
+
 - 最底层为模型层
 
   提供各种模型类型的直接调用、预定义模型配置信息、获取预定义/远程模型列表、模型凭据鉴权方法，不同模型额外提供了特殊方法，如 LLM 提供预计算 tokens 方法、获取费用信息方法等，**可横向扩展**同供应商下不同的模型（支持的模型类型下）。
@@ -73,17 +58,11 @@ Model Runtime 分三层：
 
   - 模型凭据(**在供应商层定义**)：这是一类不经常变动，一般在配置好后就不会再变动的参数，如 **api_key**、**server_url** 等。在DifyRuntime中，他们的参数名一般为**credentials: Dict[str, any]**，Provider层的credentials会直接被传递到这一层，不需要再单独定义。
 
-## 下一步
-
-### [增加新的供应商配置 👈🏻](./docs/zh_Hans/provider_scale_out.md)
-当添加后，这里将会出现一个新的供应商
-
-![Alt text](docs/zh_Hans/images/index/image-1.png)
-
-### [为已存在的供应商新增模型 👈🏻](./docs/zh_Hans/provider_scale_out.md#增加模型)
-当添加后，对应供应商的模型列表中将会出现一个新的预定义模型供用户选择，如GPT-3.5 GPT-4 ChatGLM3-6b等，而对于支持自定义模型的供应商，则不需要新增模型。
-
-![Alt text](docs/zh_Hans/images/index/image-2.png)
+
+
 
-### [接口的具体实现 👈🏻](./docs/zh_Hans/interfaces.md)
-你可以在这里找到你想要查看的接口的具体实现，以及接口的参数和返回值的具体含义。
+
+### 如果你想实现自定义的服务商模型能力
+- [移步这里 👈🏻](./docs/zh_Hans/interfaces.md)
+- [关于自定义模型的细节](./docs/zh_Hans/provider_scale_out.md)
+

model-providers/model_providers/core/model_runtime/docs/en_US/provider_scale_out.md

@@ -1,5 +1,7 @@
 ## Adding a New Provider
 
+> If you have completed the following steps, you can configure the corresponding supplier in the `model_providers.yaml` file.
+
 Providers support three types of model configuration methods:
 
 - `predefined-model` Predefined model
@@ -10,7 +12,7 @@ Providers support three types of model configuration methods:
 
   Users need to add credential configurations for each model.
 
-- `fetch-from-remote` Fetch from remote
+- `fetch-from-remote` Fetch from a remote source (Dify has not implemented this, and we do not intend to implement it either).
 
   This is consistent with the `predefined-model` configuration method. Only unified provider credentials need to be configured, and models are obtained from the provider through credential information.
 

model-providers/model_providers/core/model_runtime/docs/zh_Hans/customizable_model_scale_out.md

@@ -6,10 +6,6 @@
 
 需要注意的是，对于自定义模型，每一个模型的接入都需要填写一个完整的供应商凭据。
 
-而不同于预定义模型，自定义供应商接入时永远会拥有如下两个参数，不需要在供应商yaml中定义。
-
-![Alt text](images/index/image-3.png)
-
 
 在前文中，我们已经知道了供应商无需实现`validate_provider_credential`，Runtime会自行根据用户在此选择的模型类型和模型名称调用对应的模型层的`validate_credentials`来进行验证。
 
@@ -294,4 +290,4 @@ provider_credential_schema:
       """
   ```
 
-接口方法说明见：[Interfaces](./interfaces.md)，具体实现可参考：[llm.py](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/llm/llm.py)。
+接口方法说明见：[Interfaces](./interfaces.md)，具体实现可参考代码 

model-providers/model_providers/core/model_runtime/docs/zh_Hans/predefined_model_scale_out.md

@@ -169,4 +169,4 @@ pricing:  # 价格信息
       """
   ```
 
-接口方法说明见：[Interfaces](./interfaces.md)，具体实现可参考：[llm.py](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/llm/llm.py)。
+接口方法说明见：[Interfaces](./interfaces.md)，具体实现可参考代码 

model-providers/model_providers/core/model_runtime/docs/zh_Hans/provider_scale_out.md

@@ -1,5 +1,7 @@
 ## 增加新供应商
 
+> 如果你完成了下面步骤，即可在`model_providers.yaml`中配置对应供应商
+
 供应商支持三种模型配置方式：
 
 - `predefined-model  ` 预定义模型
@@ -10,7 +12,7 @@
 
   用户需要新增每个模型的凭据配置，如Xinference，它同时支持 LLM 和 Text Embedding，但是每个模型都有唯一的**model_uid**，如果想要将两者同时接入，就需要为每个模型配置一个**model_uid**。
 
-- `fetch-from-remote` 从远程获取
+- `fetch-from-remote` 从远程获取（Dify没有实现，我们也不会实现）
 
   与 `predefined-model` 配置方式一致，只需要配置统一的供应商凭据即可，模型通过凭据信息从供应商获取。