(Core) tool_choice overiding for with_structured_output for BaseChatModel

[keenborder786]

• Description: Giving the client the ability to override the tool_choice when using with_structured_output, this might be important for some ChatModel Interfaces like ChatLiteLLM which supports multiple LLM provider and therefore same tool choice cannot be used. This change won't break any of the existing user's use of with_structured_output.
• Issue: [ChatLiteLLM] litellm.UnsupportedParamsError: VertexAI doesn't support tool_choice=any. Supported tool_choice values=['auto', 'required', json object] #29308

Basically for above issue, with this change the user will be able to do the following:

import os
from langchain_community.chat_models import ChatLiteLLM
from langchain_google_genai import ChatGoogleGenerativeAI

from typing import Optional

from pydantic import BaseModel, Field


class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="The setup of the joke")
    punchline: str = Field(description="The punchline to the joke")
    rating: Optional[int] = Field(
        default=None, description="How funny the joke is, from 1 to 10"
    )


llm_litellm = ChatLiteLLM(
    model="gemini/gemini-2.0-flash-exp",
    api_key=os.getenv("API_KEY"),
)

stuctured_litellm = llm_litellm.with_structured_output(Joke, include_raw=True, tool_choice="any")

try:
    response_litellm = stuctured_litellm.invoke("Tell me a joke about cats")
    print(response_litellm)
except Exception as e:
    print(f"An error occured for litellm: {str(e)}")

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jan 25, 2025 9:41pm

[keenborder786]

@ccurme

[ccurme]

I'm not sure we want to put it on the user to know what value of tool_choice to use, since for structured output we know we want to force structure according to the (one) given schema.

ChateLiteLLM.bind_tools should be handling the tool_choice="any" case. It has some conditions there for Azure, does doing something similar for Gemini fix things?

[keenborder786]

@ccurme I am not so sure but will check