Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[API, Feature] Support response prefill for openai API #1490

Merged
merged 2 commits into from
Sep 22, 2024
Merged

Conversation

Ying1123
Copy link
Member

@Ying1123 Ying1123 commented Sep 22, 2024

Support response prefill for openai API like what anthropic supports:
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response

Example usage:

# Launch server
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
# Send request
import openai
from openai import OpenAI


client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {
            "role": "user",
            "content":
"""
Extract the name, size, price, and color from this product description as a JSON object:

<description>
The SmartHome Mini is a compact smart home assistant available in black or white for only $49.99. At just 5 inches wide, it lets you control lights, thermostats, and other connected devices via voice or app—no matter where you place it in your home. This affordable little hub brings convenient hands-free control to your smart devices.
</description>
""",
        },
        {
            "role": "assistant",
            "content": "{\n",
        },
    ],
    temperature=0,
)

print(response.choices[0].message.content)

Output:

"name": "SmartHome Mini",
"size": "5 inches wide",
"price": "$49.99",
"color": "black or white"
}

Without the prefilled response of "{\n", the output is:

Here's the extracted information as a JSON object:

```json
{
"name": "SmartHome Mini",
"size": "5 inches wide",
"price": "$49.99",
"color": "black or white"
}
```

If you want to store the color as an array of strings, you could modify the JSON object like this:
```json
{
"name": "SmartHome Mini",
"size": "5 inches wide",
"price": "$49.99",
"color": ["black", "white"]
}
```

@Ying1123 Ying1123 force-pushed the ying-api branch 2 times, most recently from 6c35125 to 9794d54 Compare September 22, 2024 09:58
@merrymercy merrymercy disabled auto-merge September 22, 2024 13:46
@merrymercy merrymercy merged commit e4780cf into main Sep 22, 2024
11 of 12 checks passed
@merrymercy merrymercy deleted the ying-api branch September 22, 2024 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants