From 9cb7869b318eddb6d97b7c430da7011ba2cfc4a2 Mon Sep 17 00:00:00 2001 From: Han Wang Date: Thu, 25 Apr 2024 14:24:19 +0800 Subject: [PATCH] [Doc] Add flex flow doc (#2981) # Description Please add an informative description that covers that changes made by the pull request and link all relevant issues. # All Promptflow Contribution checklist: - [ ] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [ ] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [ ] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [ ] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [ ] Pull request includes test coverage for the included changes. --------- Co-authored-by: Zhengfei Wang <38847871+zhengfeiwang@users.noreply.github.com> Co-authored-by: Clement Wang Co-authored-by: Clement Wang --- docs/README.md | 2 +- .../azureai/run-promptflow-in-azure-ai.md | 2 +- .../azureai/use-flow-in-azure-ml-pipeline.md | 4 +- docs/concepts/concept-flows.md | 15 +- .../add-conditional-control-to-a-flow.md | 0 .../develop-chat-flow.md | 0 .../develop-evaluation-flow.md | 0 .../develop-standard-flow.md | 0 .../how-to-guides/develop-a-dag-flow/index.md | 26 ++ .../init-and-test-a-flow.md | 0 .../process-image-in-flow.md | 0 ...ing-external-files-or-folders-in-a-flow.md | 0 .../develop-a-flex-flow/class-based-flow.md | 296 ++++++++++++++++++ .../function-based-flow.md | 144 +++++++++ .../develop-a-flex-flow/index.md | 26 ++ .../input-output-format.md | 43 +++ docs/how-to-guides/develop-a-flow/index.md | 14 - docs/how-to-guides/develop-a-prompty/index.md | 2 +- .../prompty-output-format.md | 2 +- ...in-flex-flow.md => use-prompty-in-flow.md} | 8 +- docs/how-to-guides/index.md | 3 +- docs/how-to-guides/quick-start.md | 10 +- .../run-and-evaluate-a-flow/index.md | 4 +- docs/index.md | 2 +- scripts/docs/conf.py | 5 +- 25 files changed, 572 insertions(+), 36 deletions(-) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/add-conditional-control-to-a-flow.md (100%) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/develop-chat-flow.md (100%) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/develop-evaluation-flow.md (100%) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/develop-standard-flow.md (100%) create mode 100644 docs/how-to-guides/develop-a-dag-flow/index.md rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/init-and-test-a-flow.md (100%) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/process-image-in-flow.md (100%) rename docs/how-to-guides/{develop-a-flow => develop-a-dag-flow}/referencing-external-files-or-folders-in-a-flow.md (100%) create mode 100644 docs/how-to-guides/develop-a-flex-flow/class-based-flow.md create mode 100644 docs/how-to-guides/develop-a-flex-flow/function-based-flow.md create mode 100644 docs/how-to-guides/develop-a-flex-flow/index.md create mode 100644 docs/how-to-guides/develop-a-flex-flow/input-output-format.md delete mode 100644 docs/how-to-guides/develop-a-flow/index.md rename docs/how-to-guides/develop-a-prompty/{use-prompty-in-flex-flow.md => use-prompty-in-flow.md} (93%) diff --git a/docs/README.md b/docs/README.md index e7f4a169822..b900ae65d62 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,7 +10,7 @@ Below is a table of important doc pages. |----------------|----------------| |Quick start|[Getting started with prompt flow](./how-to-guides/quick-start.md)| |Concepts|[Flows](./concepts/concept-flows.md)
[Tools](./concepts/concept-tools.md)
[Connections](./concepts/concept-connections.md)
[Variants](./concepts/concept-variants.md)
| -|How-to guides|[How to initialize and test a flow](./how-to-guides/develop-a-flow/init-and-test-a-flow.md)
[How to run and evaluate a flow](./how-to-guides/run-and-evaluate-a-flow/index.md)
[How to tune prompts using variants](./how-to-guides/tune-prompts-with-variants.md)
[How to deploy a flow](./how-to-guides/deploy-a-flow/index.md)
[How to create and use your own tool package](./how-to-guides/develop-a-tool/create-and-use-tool-package.md)| +|How-to guides|[How to initialize and test a flow](./how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md)
[How to run and evaluate a flow](./how-to-guides/run-and-evaluate-a-flow/index.md)
[How to tune prompts using variants](./how-to-guides/tune-prompts-with-variants.md)
[How to deploy a flow](./how-to-guides/deploy-a-flow/index.md)
[How to create and use your own tool package](./how-to-guides/develop-a-tool/create-and-use-tool-package.md)| |Tools reference|[LLM tool](./reference/tools-reference/llm-tool.md)
[Prompt tool](./reference/tools-reference/prompt-tool.md)
[Python tool](./reference/tools-reference/python-tool.md)
[Embedding tool](./reference/tools-reference/embedding_tool.md)
[SERP API tool](./reference/tools-reference/serp-api-tool.md) || diff --git a/docs/cloud/azureai/run-promptflow-in-azure-ai.md b/docs/cloud/azureai/run-promptflow-in-azure-ai.md index fe4c7f75b1f..e0bb471bcce 100644 --- a/docs/cloud/azureai/run-promptflow-in-azure-ai.md +++ b/docs/cloud/azureai/run-promptflow-in-azure-ai.md @@ -155,7 +155,7 @@ At the end of stream logs, you can find the `portal_url` of the submitted run, c ### Run snapshot of the flow with additional includes -Flows that enabled [additional include](../../how-to-guides/develop-a-flow/referencing-external-files-or-folders-in-a-flow.md) files can also be submitted for execution in the workspace. Please note that the specific additional include files or folders will be uploaded and organized within the **Files** folder of the run snapshot in the cloud. +Flows that enabled [additional include](../../how-to-guides/develop-a-dag-flow/referencing-external-files-or-folders-in-a-flow.md) files can also be submitted for execution in the workspace. Please note that the specific additional include files or folders will be uploaded and organized within the **Files** folder of the run snapshot in the cloud. ![img](../../media/cloud/azureml/run-with-additional-includes.png) diff --git a/docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md b/docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md index f1c8b9d44bb..bb87a56babf 100644 --- a/docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md +++ b/docs/cloud/azureai/use-flow-in-azure-ml-pipeline.md @@ -1,7 +1,7 @@ # Use flow in Azure ML pipeline job In practical scenarios, flows fulfill various functions. For example, consider an offline flow specifically designed to assess the relevance score for communication sessions between humans and agents. This flow is triggered nightly and processes a substantial amount of session data. In such a context, Parallel component and AzureML pipeline emerge as the optimal choices for handling large-scale, highly resilient, and efficient offline batch requirements. -Once you’ve developed and thoroughly tested your flow using the guidelines in the [init and test a flow](../../how-to-guides/develop-a-flow/init-and-test-a-flow.md) section, this guide will walk you through utilizing your flow as a parallel component within an AzureML pipeline job. +Once you’ve developed and thoroughly tested your flow, this guide will walk you through utilizing your flow as a parallel component within an AzureML pipeline job. :::{admonition} Pre-requirements To enable this feature, customer need to: @@ -329,7 +329,7 @@ Given above, if your flow has logic relying on identity or environment variable, | key | source | type | description | | ----------- | ------ | ---------------------- | ------------------------------------------------------------ | | data | fixed | uri_folder or uri_file | required; to pass in input data. Supported format includes [`mltable`](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-mltable?view=azureml-api-2&tabs=cli#authoring-mltable-files) and list of jsonl files. | -| run_outputs | fixed | uri_folder | optional; to pass in output of a standard flow for [an evaluation flow](../../how-to-guides/develop-a-flow/develop-evaluation-flow.md). Should be linked to a `flow_outputs` of a previous flow node in the pipeline. | +| run_outputs | fixed | uri_folder | optional; to pass in output of a standard flow for [an evaluation flow](../../how-to-guides/develop-a-dag-flow/develop-evaluation-flow.md). Should be linked to a `flow_outputs` of a previous flow node in the pipeline. | ### Output ports diff --git a/docs/concepts/concept-flows.md b/docs/concepts/concept-flows.md index ff58925b1a1..b3d7012f1ab 100644 --- a/docs/concepts/concept-flows.md +++ b/docs/concepts/concept-flows.md @@ -12,7 +12,7 @@ Our [examples](https://github.com/microsoft/promptflow/tree/main/examples/flex-f Thus LLM apps can be defined as Directed Acyclic Graphs (DAGs) of function calls. These DAGs are flows in prompt flow. -A flow in prompt flow is a DAG of functions (we call them [tools](./concept-tools.md)). These functions/tools connected via input/output dependencies and executed based on the topology by prompt flow executor. +A `DAG flow` in prompt flow is a DAG of functions (we call them [tools](./concept-tools.md)). These functions/tools connected via input/output dependencies and executed based on the topology by prompt flow executor. A flow is represented as a YAML file and can be visualized with our [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow). Here is an example `flow.dag.yaml`: @@ -20,6 +20,17 @@ A flow is represented as a YAML file and can be visualized with our [Prompt flow Please refer to our [examples](https://github.com/microsoft/promptflow/tree/main/examples/flows) to learn how to write a `DAG flow`. +## When to use Flex or DAG flow + +`Dag flow` provides a UI-friendly way to develop your LLM app, which has the following benifits: +- **Low code**: user can drag-and-drop in UI to create a LLM app. +- **DAG Visualization**: user can easily understand the logic structure of the app with DAG view. + +`Flex flow` provides a code-friendly way to develop your LLM app, which has the following benifits: +- **Quick start**: Users can quickly test with a simple prompt, then customize with python code with Tracing visualization UI. +- **More advanced orchestration**: Users can write complex flow with Python built-in control operators (if-else, foreach) or other 3rd party / open-source library. +- **Easy onboard from other platforms**: user might already onboard platforms like `langchain` and `sematic kernel` with existing code. User can easily onboard promptflow with a few code changes. + ## Flow types Prompt flow examples organize flows by three categories: @@ -42,6 +53,6 @@ DAG flow [examples](https://github.com/microsoft/promptflow/tree/main/examples/f ## Next steps - [Quick start](../how-to-guides/quick-start.md) -- [Initialize and test a flow](../how-to-guides/develop-a-flow/init-and-test-a-flow.md) +- [Initialize and test a flow](../how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md) - [Run and evaluate a flow](../how-to-guides/run-and-evaluate-a-flow/index.md) - [Tune prompts using variants](../how-to-guides/tune-prompts-with-variants.md) \ No newline at end of file diff --git a/docs/how-to-guides/develop-a-flow/add-conditional-control-to-a-flow.md b/docs/how-to-guides/develop-a-dag-flow/add-conditional-control-to-a-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/add-conditional-control-to-a-flow.md rename to docs/how-to-guides/develop-a-dag-flow/add-conditional-control-to-a-flow.md diff --git a/docs/how-to-guides/develop-a-flow/develop-chat-flow.md b/docs/how-to-guides/develop-a-dag-flow/develop-chat-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/develop-chat-flow.md rename to docs/how-to-guides/develop-a-dag-flow/develop-chat-flow.md diff --git a/docs/how-to-guides/develop-a-flow/develop-evaluation-flow.md b/docs/how-to-guides/develop-a-dag-flow/develop-evaluation-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/develop-evaluation-flow.md rename to docs/how-to-guides/develop-a-dag-flow/develop-evaluation-flow.md diff --git a/docs/how-to-guides/develop-a-flow/develop-standard-flow.md b/docs/how-to-guides/develop-a-dag-flow/develop-standard-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/develop-standard-flow.md rename to docs/how-to-guides/develop-a-dag-flow/develop-standard-flow.md diff --git a/docs/how-to-guides/develop-a-dag-flow/index.md b/docs/how-to-guides/develop-a-dag-flow/index.md new file mode 100644 index 00000000000..162d99dd855 --- /dev/null +++ b/docs/how-to-guides/develop-a-dag-flow/index.md @@ -0,0 +1,26 @@ +# Develop a dag flow + +LLM apps can be defined as Directed Acyclic Graphs (DAGs) of function calls. These DAGs are flows in prompt flow. + +A `DAG flow` in prompt flow is a DAG of functions (we call them [tools](../../concepts//concept-tools.md)). These functions/tools connected via input/output dependencies and executed based on the topology by prompt flow executor. + +A flow is represented as a YAML file and can be visualized with our [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow). Here is an example `flow.dag.yaml`: + +![flow_dag](../../media/how-to-guides/quick-start/flow_dag.png) + +Please refer to our [examples](https://github.com/microsoft/promptflow/tree/main/examples/flows) and guides in this section to learn how to write a `DAG flow`. + +Note: +- promptflow also support user develop a a flow using code. learn more on comparasion of these two [flow concepts](../../concepts/concept-flows.md). + +```{toctree} +:maxdepth: 1 + +init-and-test-a-flow +develop-standard-flow +develop-chat-flow +develop-evaluation-flow +add-conditional-control-to-a-flow +process-image-in-flow +referencing-external-files-or-folders-in-a-flow +``` \ No newline at end of file diff --git a/docs/how-to-guides/develop-a-flow/init-and-test-a-flow.md b/docs/how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/init-and-test-a-flow.md rename to docs/how-to-guides/develop-a-dag-flow/init-and-test-a-flow.md diff --git a/docs/how-to-guides/develop-a-flow/process-image-in-flow.md b/docs/how-to-guides/develop-a-dag-flow/process-image-in-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/process-image-in-flow.md rename to docs/how-to-guides/develop-a-dag-flow/process-image-in-flow.md diff --git a/docs/how-to-guides/develop-a-flow/referencing-external-files-or-folders-in-a-flow.md b/docs/how-to-guides/develop-a-dag-flow/referencing-external-files-or-folders-in-a-flow.md similarity index 100% rename from docs/how-to-guides/develop-a-flow/referencing-external-files-or-folders-in-a-flow.md rename to docs/how-to-guides/develop-a-dag-flow/referencing-external-files-or-folders-in-a-flow.md diff --git a/docs/how-to-guides/develop-a-flex-flow/class-based-flow.md b/docs/how-to-guides/develop-a-flex-flow/class-based-flow.md new file mode 100644 index 00000000000..5f73e0a9248 --- /dev/null +++ b/docs/how-to-guides/develop-a-flex-flow/class-based-flow.md @@ -0,0 +1,296 @@ +# Class based flow + +:::{admonition} Experimental feature +This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental). +::: + +When user need to persist objects (like connection) in memory during multiple rounds of flow runs, they can write a callable class as flow entry and put persist params in `__init__` method. + +If user need to log metrics on batch run outputs, they can add an `__aggregate__` method and it will be scheduled after batch run finishes. +The `__aggregate__` method should only contain 1 params which is list of batch run results. + +See [connection support](#connection-support) & [aggregation support](#aggregation-support) for more details. + +## Class as a flow + +Assume we have a file `flow_entry.py`: + +```python +class Reply(TypedDict): + output: str + +class MyFlow: + def __init__(self, model_config: AzureOpenAIModelConfiguration, flow_config: dict): + """Flow initialization logic goes here.""" + self.model_config = model_config + self.flow_config = flow_config + + def __call__(question: str) -> Reply: + """Flow execution logic goes here.""" + return Reply(output=output) + + def __aggregate__(self, line_results: List[str]) -> dict: + """Aggregation logic goes here. Return key-value pair as metrics.""" + return {"key": val} +``` + + +## Flow test + +Since flow's definition is function/callable class. We recommend user directly run it like running other scripts: + +```python +class MyFlow: + pass +if __name__ == "__main__": + flow = MyFlow(model_config, flow_config) + output = flow(question) + metrics = flow.__aggregate__([output]) + # check metrics here +``` + +You can also test the flow using CLI: +```bash +# flow entry syntax: path.to.module:ClassName +pf flow test --flow flow_entry:MyFlow --inputs question="What's the capital of France?" --init init.json +``` + +Check out a full example here: [basic-chat](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/basic-chat) + +### Chat with a flow + +Chat with flow in CLI is supported: + +```bash +pf flow test --flow flow_entry:MyFlow --inputs inputs.json --init init.json --ui +``` + +Check [here](../chat-with-a-flow/index.md) for more information. + +## Batch run + +User can also batch run a flow. + +::::{tab-set} +:::{tab-item} CLI +:sync: CLI + +```bash +pf run create --flow "path.to.module:ClassName" --data "./data.jsonl" +``` + +::: + +:::{tab-item} SDK +:sync: SDK +```python +# user can also directly use entry in `flow` param for batch run +pf.run(flow="path.to.module:ClassName", init="./init.jsonl", data="./data.jsonl") +``` + +::: +:::: + +Or directly run the imported flow class or flow instance. + +```python +class MyFlow: + pass +pf.run(flow=MyFlow, init={"model_config": config, "flow_config": {}}, data="./data.jsonl") +# or +flow_obj = MyFlow(model_config=config, flow_config={}) +pf.run(flow=flow_obj, data="./data.jsonl") +``` + +Learn more on this topic on [Run and evaluate a flow](../run-and-evaluate-a-flow/index.md) + +## Define a flow yaml + +User can write a YAML file with name `flow.flex.yaml` manually or save a function/callable entry to YAML file. +This is required for advanced scenario like deployment or run in cloud. +A flow YAML may look like this: + +```yaml +$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json +entry: path.to.module:ClassName +``` + +## Batch run with YAML + +User can batch run a flow. Flow init function's param is supported by `init` parameter. + +::::{tab-set} +:::{tab-item} CLI +:sync: CLI + +User need to write an JSON file as init's value since it's hard to write model config in command line. + +```json +{ + "model_config": { + "azure_endpoint": "my_endpoint", + "azure_deployment": "my_deployment", + "api_key": "actual_api_key" + }, + "flow_config": {} +} +``` + +```bash +pf run create --flow "./flow.flex.yaml" --data "./data.jsonl" --init init.json +``` + +::: + +:::{tab-item} SDK +:sync: SDK + +```python +pf = PFClient() + +config = AzureOpenAIModelConfiguration( + azure_deployment="my_deployment", + api_key="actual_key" +) +# if init's value is not json serializable, raise user error +pf.run(flow="./flow.flex.yaml", init={"model_config": config, "flow_config": {}}, data="./data.jsonl") + +# when submit to cloud, user can only use connection +# in runtime executor will resolve connection in AzureOpenAIModelConfiguration and set connection's fields to ModelConfig: equal to original ModelConfiguration.from_connection() +config = AzureOpenAIModelConfiguration( + azure_deployment="my_embedding_deployment", + connection="my-aoai-connection", +) +pfazure.run(flow="./flow.flex.yaml", init={"model_config": config, "flow_config": {}}, data="./data.jsonl") +``` + +::: +:::: + +## Deploy a flow + +User can serve a flow. Flow init function's param is supported by `init` parameter. +The flow should have complete init/inputs/outputs specification in YAML to make sure serving swagger can be generated. + +User need to write an JSON file as init's value since it's hard to write model config in command line. + +```json +{ + "model_config": { + "azure_endpoint": "my_endpoint", + "azure_deployment": "my_deployment", + "api_key": "actual_api_key" + }, + "flow_config": {} +} +``` + +```bash +# user can only pass model config by file +pf flow serve --source "./" --port 8088 --host localhost --init path/to/init.json +``` + +Learn more: [Deploy a flow](../deploy-a-flow/index.md). + +## Connection support + +### Model config in `__init__` + +Just like example in [batch run](#batch-run-with-yaml), it's supported to reference connection in ModelConfig. +And connection will be resolved and flatten connection's fields to ModelConfig. + +### Connection in `__init__` + +It's also supported to directly pass connection by **name** in `__init__`. + +```python +class MyFlow: + def __init__(self, my_connection: AzureOpenAIConnection): + pass +``` + +Note: + +- Union of connection types(`Union[OpenAIConnection, AzureOpenAIConnection]`) is not supported. + +#### Batch run with connection + +User can pass connection name to connection field in `init`. + +In local, the connection name will be replaced with local connection object in execution time. +In cloud, the connection name will be replaced with workspace's connection object in execution time. + +```python +# local connection "my_connection"'s instance will be passed to `__init__` +pf.run(flow="./flow.flex.yaml", init={"connection": "my_connection"}, data="./data.jsonl") +# cloud connection "my_cloud_connection"'s instance will be passed to `__init__` +pfazure.run(flow="./flow.flex.yaml", init={"connection": "my_cloud_connection"}, data="./data.jsonl") +``` + +### Environment variable connections(EVC) + +If flow YAML has `environment_variables` and it's value is a connection reference like this: + +```yaml +environment_variables: + AZURE_OPENAI_API_KEY: ${open_ai_connection.api_key} + AZURE_OPENAI_ENDPOINT: ${open_ai_connection.api_base} +``` + +The environment variable's value will be resolved to actual value in runtime. +If the connection not exist (in local or cloud), connection not found error will be raised. + +**Note**: User can override the `environment_variables` with existing environment variable keys in `flow.flex.yaml`: + +```bash +pf run create --flow . --data ./data.jsonl --environment-variables AZURE_OPENAI_API_KEY='${new_connection.api_key}' AZURE_OPENAI_ENDPOINT='my_endpoint' +``` + +Overriding with environment variable names which not exist in `flow.flex.yaml` is not supported. +Which means if user added environment variables which does not exist in `flow.flex.yaml` in runtime, it's value won't be resolved. + +For example, + +```bash +pf run create --flow . --data ./data.jsonl --environment-variables NEW_API_KEY='${my_new_connection.api_key}' +``` + +The `NEW_API_KEY`'s value won't be resolved to connection's API key. + +## Aggregation support + +Aggregation support is introduce to help user calculate metrics. + +```python +class MyFlow: + def __call__(text: str) -> str: + """Flow execution logic goes here.""" + pass + + # will only execute once after batch run finished. + # the processed_results will be list of __call__'s output and we will log the return value as metrics automatically. + def __aggregate__(self, processed_results: List[str]) -> dict: + for element in processed_results: + # If __call__'s output is primitive type, element will be primitive type. + # If __call__'s output is dataclass, element will be a dictionary, but can access it's attribute with `element.attribute_name` + # For other cases, it's recommended to access by key `element["attribute_name"]` + +``` + +**Note**: + +There's several limitations on aggregation support: + +- The aggregation function will only execute in batch run. +- Only 1 hard coded `__aggregate__` function is supported. +- The `__aggregate__` will only be passed **1** positional arguments when executing. +- The aggregation function’s input will be flow run’s outputs list. + - Each element inside `processed_results` passed passed inside `__aggregate__` function is not same object with each line's `__call__` returns. + - The reconstructed element is a dictionary which supports 1 layer attribute access. But it's recommended to access them by key. See the above example for usage. +- If aggregation function accept more than 1 arguments, raise error in submission phase. + +## Next steps + +- [Input output format](./input-output-format.md) +- [Class based flow sample](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/chat-basic/README.md) +- [Class based flow evaluation sample](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/eval-code-quality/README.md) diff --git a/docs/how-to-guides/develop-a-flex-flow/function-based-flow.md b/docs/how-to-guides/develop-a-flex-flow/function-based-flow.md new file mode 100644 index 00000000000..2283b5d6b87 --- /dev/null +++ b/docs/how-to-guides/develop-a-flex-flow/function-based-flow.md @@ -0,0 +1,144 @@ +# Function based flow + +:::{admonition} Experimental feature +This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental). +::: + +User can directly use a function as flow entry. + +## Function as a flow + +Assume we have a file `flow_entry.py`: + +```python +from promptflow.tracing import trace + +class Reply(TypedDict): + output: str + +@trace +def my_flow(question: str) -> Reply: + # flow logic goes here + pass +``` + +**Note** function decorated with `@trace` will emit trace can be viewed in UI provided by PromptFlow. Check [here](../tracing/index.md) for more information. + +## Flow test + +Since flow's definition is normal python function/callable class. We recommend user directly run it like running other scripts: + +```python +from flow_entry import my_flow + +if __name__ == "__main__": + output = my_flow(question="What's the capital of France?") + print(output) +``` + +You can also test the flow using CLI: +```bash +# flow entry syntax: path.to.module:function_name +pf flow test --flow flow_entry:my_flow --inputs question="What's the capital of France?" +``` + +Check out a full example here: [basic](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/basic) + +### Chat with a flow + +Start a UI to chat with a flow: + +```bash +pf flow test --flow flow_entry:my_flow --inputs question="What's the capital of France?" --ui +``` + +Check [here](../chat-with-a-flow/index.md) for more information. + +## Batch run + +User can also batch run a flow. + +::::{tab-set} +:::{tab-item} CLI +:sync: CLI + +```bash +pf run create --flow "path.to.module:function_name" --data "./data.jsonl" +``` + +::: + +:::{tab-item} SDK +:sync: SDK +```python + +from path.to.module import my_flow +pf.run(flow=my_flow, data="./data.json;") + +# user can also directly use entry in `flow` param for batch run +pf.run(flow="path.to.module:function_name", data="./data.jsonl") +``` +::: +:::: + +Learn more on this topic on [Run and evaluate a flow](../run-and-evaluate-a-flow/index.md) + +## Define a flow yaml + +User can write a YAML file with name `flow.flex.yaml` manually or save a function/callable entry to YAML file. +This is required for advanced scenario like deployment or run in cloud. +A flow YAML may look like this: + +```yaml +$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json +entry: path.to.module:function_name +sample: + question: "what's the capital of France?" +``` + +## Batch run with YAML + +User can batch run a flow with YAML. + +::::{tab-set} +:::{tab-item} CLI +:sync: CLI + +```bash +# against flow file +pf run create --flow "path/to/flow/flow.flex.yaml" --data "./data.jsonl" +# against a folder if it has a flow.flex.yaml file +pf run create --flow "path/to/flow" --data "./data.jsonl" +``` + +::: + +:::{tab-item} SDK +:sync: SDK + +```python +pf = PFClient() +pf.run(flow="./flow.flex.yaml", data="./data.jsonl") +``` + +::: +:::: + +## Deploy a flow + +User can serve a flow as a http endpoint locally or deploy it to multiple platforms. + +```bash +# serve locally from a folder if it has a flow.flex.yaml file +pf flow serve --source "path/to/flow/dir" --port 8088 --host localhost + +# serve locally from certain file +pf flow serve --source "./flow.flex.yaml" --port 8088 --host localhost +``` +Learn more: [Deploy a flow](../deploy-a-flow/index.md). + +## Next steps + +- [Class based flow](./class-based-flow.md) +- [Input output format](./input-output-format.md) +- [Function based flow sample](https://github.com/microsoft/promptflow/blob/main/examples/flex-flows/basic/README.md) diff --git a/docs/how-to-guides/develop-a-flex-flow/index.md b/docs/how-to-guides/develop-a-flex-flow/index.md new file mode 100644 index 00000000000..b5db3e920ac --- /dev/null +++ b/docs/how-to-guides/develop-a-flex-flow/index.md @@ -0,0 +1,26 @@ +# Develop a flow + +:::{admonition} Experimental feature +This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental). +::: + +You can create LLM apps using a Python function or class as the entry point, which encapsulating your app logic. You can directly test or run these entries with pure code experience. + +In PromptFlow, these functions or classes are referred to as `flow` or `flex flow`. + +Alternatively, you can define a `flow.flex.yaml` that points to these entries (`entry:function_name` or `entry:ClassName`). This enables testing, running, or viewing traces via the [Promptflow VS Code Extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow). + +Our [examples](https://github.com/microsoft/promptflow/tree/main/examples/flex-flows) should give you a good idea on how to write flows. + +Note: +- The term *Flex* is a shorthand for *flexible*, indicating its adaptability to most scenarios with minimal adjustments. +- PromptFlow also supports the development of a `dag flow`. learn more on comparasion of these two [flow concepts](../../concepts/concept-flows.md). + + +```{toctree} +:maxdepth: 1 + +function-based-flow +class-based-flow +input-output-format +``` diff --git a/docs/how-to-guides/develop-a-flex-flow/input-output-format.md b/docs/how-to-guides/develop-a-flex-flow/input-output-format.md new file mode 100644 index 00000000000..bb7356494c1 --- /dev/null +++ b/docs/how-to-guides/develop-a-flex-flow/input-output-format.md @@ -0,0 +1,43 @@ +# Input output format + +:::{admonition} Experimental feature +This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental). +::: + +## Supported types + +Promptflow officially support below types in flow. + +- Inputs: primitive types(`int`, `float`, `bool`, `str`), `dict`, `TypedDict`, `list` + +- Outputs: primitive types(`int`, `float`, `bool`, `str`), `dict`, `TypedDict`, `dataclass`, `list` + +- Init: primitive types(`int`, `float`, `bool`, `str`), `Connection`, `ModelConfiguration`, `TypedDict`, `list` + +If user has non-supported types in code/YAML, validation error will be raised. + +```python +# using unsupported types in flow will fail with validation error +class MyOwnClass: + pass + +class MyFlow: + # not supported + def __init__(self, my_own_obj: MyOwnClass): + pass + +# not supported +def my_flow(my_own_obj: MyOwnClass): + pass +``` + +Sample validation error: "The input 'my_own_obj' is of a complex python type. Please use a dict instead." + + + +## Stream + +Stream is supported in flow, you just need to return a generator type in your function. +Reference openai doc on how to do it using plain python code: [how_to_stream_completions](https://cookbook.openai.com/examples/how_to_stream_completions). + +Reference this flow [sample](https://microsoft.github.io/promptflow/tutorials/stream-flex-flow.html) for details. \ No newline at end of file diff --git a/docs/how-to-guides/develop-a-flow/index.md b/docs/how-to-guides/develop-a-flow/index.md deleted file mode 100644 index eb23ee252b8..00000000000 --- a/docs/how-to-guides/develop-a-flow/index.md +++ /dev/null @@ -1,14 +0,0 @@ -# Develop a flow -We provide guides on how to develop a flow by writing a flow yaml from scratch in this section. - -```{toctree} -:maxdepth: 1 - -init-and-test-a-flow -develop-standard-flow -develop-chat-flow -develop-evaluation-flow -add-conditional-control-to-a-flow -process-image-in-flow -referencing-external-files-or-folders-in-a-flow -``` \ No newline at end of file diff --git a/docs/how-to-guides/develop-a-prompty/index.md b/docs/how-to-guides/develop-a-prompty/index.md index 99973117cc9..c9bc1790998 100644 --- a/docs/how-to-guides/develop-a-prompty/index.md +++ b/docs/how-to-guides/develop-a-prompty/index.md @@ -416,5 +416,5 @@ The trace UI will record the execution details of each line in the data file, pr :hidden: prompty-output-format -use-prompty-in-flex-flow +use-prompty-in-flow ``` \ No newline at end of file diff --git a/docs/how-to-guides/develop-a-prompty/prompty-output-format.md b/docs/how-to-guides/develop-a-prompty/prompty-output-format.md index bd0fedc1746..e27925d6f6c 100644 --- a/docs/how-to-guides/develop-a-prompty/prompty-output-format.md +++ b/docs/how-to-guides/develop-a-prompty/prompty-output-format.md @@ -66,7 +66,7 @@ Prompty can return the content of the first choice as a dictionary object when t - The `response_format` is defined as `type: json_object` in the parameters - The template specifies the JSON format for the return value. -**Note**: `response_format` is compatible with `GPT-4 Turbo` and all GPT-3.5 Turbo models newer than `gpt-3.5-turbo-1106`. For more details, refer to this [document](https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format). +**Note**: `json_object` response_format is compatible with `GPT-4 Turbo` and all GPT-3.5 Turbo models newer than `gpt-3.5-turbo-1106`. For more details, refer to this [document](https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format). Here’s how to configure a prompty for JSON object output: ```yaml diff --git a/docs/how-to-guides/develop-a-prompty/use-prompty-in-flex-flow.md b/docs/how-to-guides/develop-a-prompty/use-prompty-in-flow.md similarity index 93% rename from docs/how-to-guides/develop-a-prompty/use-prompty-in-flex-flow.md rename to docs/how-to-guides/develop-a-prompty/use-prompty-in-flow.md index efe93ba81a9..9c649e38c9c 100644 --- a/docs/how-to-guides/develop-a-prompty/use-prompty-in-flex-flow.md +++ b/docs/how-to-guides/develop-a-prompty/use-prompty-in-flow.md @@ -1,10 +1,10 @@ -# Using prompty in flex flow +# Using prompty in flow :::{admonition} Experimental feature This is an experimental feature, and may change at any time. Learn [more](../faq.md#stable-vs-experimental). ::: -Because Prompty can be called as a function, user can use prompty in a `flex flow` which is can be a python function or class. +Because Prompty can be called as a function, user can use prompty in a `flow` which is can be a python function or class. This allows user to do more customization logic with prompty. @@ -104,9 +104,9 @@ User can run above code as normal python file. python path/to/entry.py ``` -## Test the class as a flex flow +## Test the class as a flow -User can also leverage promptflow to test the class as a `flex flow`. +User can also leverage promptflow to test the class as a `flow`. ```bash pf flow test --flow file:ChatFlow --init init.json --inputs "question=What is ChatGPT?" diff --git a/docs/how-to-guides/index.md b/docs/how-to-guides/index.md index 7bd1036a7e4..3eebb29671f 100644 --- a/docs/how-to-guides/index.md +++ b/docs/how-to-guides/index.md @@ -17,7 +17,8 @@ develop-a-prompty/index ```{toctree} :caption: Flow :maxdepth: 1 -develop-a-flow/index +develop-a-flex-flow/index +develop-a-dag-flow/index execute-flow-as-a-function chat-with-a-flow/index run-and-evaluate-a-flow/index diff --git a/docs/how-to-guides/quick-start.md b/docs/how-to-guides/quick-start.md index ce51e3e4731..ae3a331b3ba 100644 --- a/docs/how-to-guides/quick-start.md +++ b/docs/how-to-guides/quick-start.md @@ -108,7 +108,7 @@ inputs: default: https://play.google.com/store/apps/details?id=com.twitter.android ... ``` -See more details of this topic in [Develop a flow](./develop-a-flow/index.md). +See more details of this topic in [Develop a flow](./develop-a-dag-flow/index.md). ### Create necessary connections @@ -288,14 +288,14 @@ Click the run flow button on the top of the visual editor to trigger flow test. :::: -See more details of this topic in [Initialize and test a flow](./develop-a-flow/init-and-test-a-flow.md). +See more details of this topic in [Initialize and test a flow](./develop-a-dag-flow/init-and-test-a-flow.md). ## Next steps Learn more on how to: -- [Develop a flow](./develop-a-flow/index.md): details on how to develop a flow by writing a flow yaml from scratch. -- [Initialize and test a flow](./develop-a-flow/init-and-test-a-flow.md): details on how develop a flow from scratch or existing code. -- [Add conditional control to a flow](./develop-a-flow/add-conditional-control-to-a-flow.md): how to use activate config to add conditional control to a flow. +- [Develop a flow](./develop-a-dag-flow/index.md): details on how to develop a flow by writing a flow yaml from scratch. +- [Initialize and test a flow](./develop-a-dag-flow/init-and-test-a-flow.md): details on how develop a flow from scratch or existing code. +- [Add conditional control to a flow](./develop-a-dag-flow/add-conditional-control-to-a-flow.md): how to use activate config to add conditional control to a flow. - [Run and evaluate a flow](./run-and-evaluate-a-flow/index.md): run and evaluate the flow using multi line data file. - [Deploy a flow](./deploy-a-flow/index.md): how to deploy the flow as a web app. - [Manage connections](./manage-connections.md): how to manage the endpoints/secrets information to access external services including LLMs. diff --git a/docs/how-to-guides/run-and-evaluate-a-flow/index.md b/docs/how-to-guides/run-and-evaluate-a-flow/index.md index 441fd124b30..b22b8defac6 100644 --- a/docs/how-to-guides/run-and-evaluate-a-flow/index.md +++ b/docs/how-to-guides/run-and-evaluate-a-flow/index.md @@ -1,6 +1,6 @@ # Run and evaluate a flow -After you have developed and tested the flow in [init and test a flow](../develop-a-flow/init-and-test-a-flow.md), this guide will help you learn how to run a flow with a larger dataset and then evaluate the flow you have created. +After you have developed and tested the flow in [init and test a flow](../develop-a-dag-flow/init-and-test-a-flow.md), this guide will help you learn how to run a flow with a larger dataset and then evaluate the flow you have created. ## Create a batch run @@ -113,7 +113,7 @@ We also have a more detailed documentation [Manage runs](./manage-runs.md) demo ## Evaluate your flow -You can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score. Please refer to [Develop evaluation flow](../develop-a-flow/develop-evaluation-flow.md) to learn how to develop an evaluation flow. +You can use an evaluation method to evaluate your flow. The evaluation methods are also flows which use Python or LLM etc., to calculate metrics like accuracy, relevance score. Please refer to [Develop evaluation flow](../develop-a-dag-flow/develop-evaluation-flow.md) to learn how to develop an evaluation flow. In this guide, we use [eval-classification-accuracy](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-classification-accuracy) flow to evaluate. This is a flow illustrating how to evaluate the performance of a classification system. It involves comparing each prediction to the groundtruth and assigns a `Correct` or `Incorrect` grade, and aggregating the results to produce metrics such as `accuracy`, which reflects how good the system is at classifying the data. diff --git a/docs/index.md b/docs/index.md index 05ea406c572..0c202fb9594 100644 --- a/docs/index.md +++ b/docs/index.md @@ -39,7 +39,7 @@ This documentation site contains guides for prompt flow [sdk, cli](https://pypi. - header: "📒 How-to Guides" content: " Articles guide user to complete a specific task in prompt flow.

- - [Develop a flow](how-to-guides/develop-a-flow/index.md)
+ - [Develop a flow](how-to-guides/develop-a-flex-flow/index.md)
- [Run and evaluate a flow](how-to-guides/run-and-evaluate-a-flow/index.md)
- [Develop custom tool](how-to-guides/develop-a-tool/create-and-use-tool-package.md)
- [Deploy a flow](how-to-guides/deploy-a-flow/index.md)
diff --git a/scripts/docs/conf.py b/scripts/docs/conf.py index 5b47fb08b02..4900c87f159 100644 --- a/scripts/docs/conf.py +++ b/scripts/docs/conf.py @@ -66,7 +66,10 @@ "deploy-using-docker.html", "deploy-using-kubernetes.html", "https://portal.azure.com/#create/Microsoft.CognitiveServicesTextAnalytics", # sphinx recognizes #create as an anchor while it's not. # noqa: E501 - "https://ms.portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/searchQuery/machine%20learning", # noqa: E501 + "https://ms.portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/searchQuery/machine%20learning", # noqa: E501, + # TODO(wanhan): update this link to sample + "https://microsoft.github.io/promptflow/tutorials/stream-flex-flow.html", + "https://github.com/microsoft/promptflow/tree/main/examples/flex-flows/basic-chat", ] linkcheck_exclude_documents = [