Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure SDG can take advantage of vLLM's guided decoding #527

Open
bbrowning opened this issue Jan 29, 2025 · 1 comment
Open

Ensure SDG can take advantage of vLLM's guided decoding #527

bbrowning opened this issue Jan 29, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@bbrowning
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We have some upcoming work, such as annotations, that requires structured output from vLLM. Here's an example of the type of vLLM parameters we need to be able to pass from a pipeline into the vLLM backend to enable guided decoding. The key part is the guided_choice section there, where we need to ensure that gets passed into vLLM to enable guided_choice decoding as documented at https://docs.vllm.ai/en/latest/features/structured_outputs.html

version: "1.0"
blocks:
  - name: gen_responses
    type: LLMBlock
    config:
      config_path: detailed_description_icl.yaml
      output_cols:
        - output
    gen_kwargs:
      max_tokens: 5
      temperature: 0
      extra_body:
        guided_choice:
          - "joy"
          - "sadness"
          - "anger"
          - "fear"
          - "love"

Describe the solution you'd like

Pipelines defined similar to above should cause vLLM to enable its guided_choice to coerce the responses into one of the defined possibilities.

@bbrowning bbrowning added the enhancement New feature or request label Jan 29, 2025
@bbrowning
Copy link
Contributor Author

This is similar to #481, but I decided on a narrow new issue here just to ensure a user can choose to use guided decoding in their pipeline. 481 is a bit broader in scope, including actually using it within SDG as opposed to just enabling a user to create a pipeline that uses guided decoding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant