Skip to content

Latest commit

 

History

History
225 lines (180 loc) · 10.2 KB

llm-tool.md

File metadata and controls

225 lines (180 loc) · 10.2 KB

LLM

Introduction

Prompt flow LLM tool enables you to leverage widely used large language models like OpenAI, Azure OpenAI (AOAI), and models in Azure AI Studio model catalog for natural language processing.

Note

The previous version of the LLM tool is now being deprecated. Please upgrade to latest promptflow-tools package to consume new llm tools.

Prompt flow provides a few different LLM APIs:

  • Completion: OpenAI's completion models generate text based on provided prompts.
  • Chat: OpenAI's chat models facilitate interactive conversations with text-based inputs and responses.

Prerequisite

Create OpenAI resources, Azure OpenAI resources or MaaS deployment with the LLM models (e.g.: llama2, mistral, cohere etc.) in Azure AI Studio model catalog:

  • OpenAI

    Sign up account OpenAI website

    Login and Find personal API key

  • Azure OpenAI (AOAI)

    Create Azure OpenAI resources with instruction

  • MaaS deployment

    Create MaaS deployment for models in Azure AI Studio model catalog with instruction

    You can create serverless connection to use this MaaS deployment.

Connections

Setup connections to provisioned resources in prompt flow.

Type Name API KEY API BASE API Type API Version
OpenAI Required Required - - -
AzureOpenAI Required Required Required Required Required
Serverless Required Required Required - -

Inputs

Text Completion

Name Type Description Required
prompt string text prompt that the language model will complete Yes
model, deployment_name string the language model to use Yes
max_tokens integer the maximum number of tokens to generate in the completion. Default is 16. No
temperature float the randomness of the generated text. Default is 1. No
stop list the stopping sequence for the generated text. Default is null. No
suffix string text appended to the end of the completion No
top_p float the probability of using the top choice from the generated tokens. Default is 1. No
logprobs integer the number of log probabilities to generate. Default is null. No
echo boolean value that indicates whether to echo back the prompt in the response. Default is false. No
presence_penalty float value that controls the model's behavior with regards to repeating phrases. Default is 0. No
frequency_penalty float value that controls the model's behavior with regards to generating rare phrases. Default is 0. No
best_of integer the number of best completions to generate. Default is 1. No
logit_bias dictionary the logit bias for the language model. Default is empty dictionary. No

Chat

Name Type Description Required
prompt string text prompt that the language model will response Yes
model, deployment_name string the language model to use Yes
max_tokens integer the maximum number of tokens to generate in the response. Default is inf. No
temperature float the randomness of the generated text. Default is 1. No
stop list the stopping sequence for the generated text. Default is null. No
top_p float the probability of using the top choice from the generated tokens. Default is 1. No
presence_penalty float value that controls the model's behavior with regards to repeating phrases. Default is 0. No
frequency_penalty float value that controls the model's behavior with regards to generating rare phrases. Default is 0. No
logit_bias dictionary the logit bias for the language model. Default is empty dictionary. No
tool_choice object value that controls which tool is called by the model. Default is null. No
tools list a list of tools the model may generate JSON inputs for. Default is null. No
response_format object an object specifying the format that the model must output. Default is null. No

Outputs

Return Type Description
string The text of one predicted completion or response of conversation

How to use LLM Tool?

  1. Setup and select the connections to OpenAI resources
  2. Configure LLM model api and its parameters
  3. Prepare the Prompt with guidance.

How to write a chat prompt?

To grasp the fundamentals of creating a chat prompt, begin with this section for an introductory understanding of jinja.

We offer a method to distinguish between different roles in a chat prompt, such as "system", "user", "assistant" and "tool". The "system", "user", "assistant" roles can have "name" and "content" properties. The "tool" role, however, should have "tool_call_id" and "content" properties. For an example of a tool chat prompt, please refer to Sample 3.

Sample 1

# system:
You are a helpful assistant.

{% for item in chat_history %}
# user:
{{item.inputs.question}}
# assistant:
{{item.outputs.answer}}
{% endfor %}

# user:
{{question}}

In LLM tool, the prompt is transformed to match the openai messages structure before sending to openai chat API.

[
    {
        "role": "system",
        "content": "You are a helpful assistant."
    },
    {
        "role": "user",
        "content": "<question-of-chat-history-round-1>"
    },
    {
        "role": "assistant",
        "content": "<answer-of-chat-history-round-1>"
    },
    ...
    {
        "role": "user",
        "content": "<question>"
    }
]

Sample 2

# system:
{# For role naming customization, the following syntax is used #}
## name:
Alice
## content:
You are a bot can tell good jokes.

In LLM tool, the prompt is transformed to match the openai messages structure before sending to openai chat API.

[
    {
        "role": "system",
        "name": "Alice",
        "content": "You are a bot can tell good jokes."
    }
]

Sample 3

This sample illustrates how to write a tool chat prompt.

# system:
You are a helpful assistant.

# user:
What is the current weather like in Boston?

# assistant:
{# The assistant message with 'tool_calls' must be followed by messages with role 'tool'. #}
## tool_calls:
{{llm_output.tool_calls}}

# tool:
{#
Messages with role 'tool' must be a response to a preceding message with 'tool_calls'.
Additionally, 'tool_call_id's should match ids of assistant message 'tool_calls'.
#}
## tool_call_id:
{{llm_output.tool_calls[0].id}}
## content:
{{tool-answer-of-last-question}}

# user:
{{question}}

In LLM tool, the prompt is transformed to match the openai messages structure before sending to openai chat API.

[
    {
        "role": "system",
        "content": "You are a helpful assistant."
    },
    {
        "role": "user",
        "content": "What is the current weather like in Boston?"
    },
    {
        "role": "assistant",
        "content": null,
        "function_call": null,
        "tool_calls": [
            {
                "id": "<tool-call-id-of-last-question>",
                "type": "function",
                "function": "<function-to-call-of-last-question>"
            }
        ]
    },
    {
        "role": "tool",
        "tool_call_id": "<tool-call-id-of-last-question>",
        "content": "<tool-answer-of-last-question>"
    }
    ...
    {
        "role": "user",
        "content": "<question>"
    }
]