How to get OpenAI with_structured_output token usage with Pydantic Schema

[zahidekilic]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import TypedDict
from langchain_openai import ChatOpenAI

# Define the structured output schema
class UserInput(TypedDict):
    info_request: str
    analysis_request: str
    other: str

# Initialize the ChatOpenAI instance
co = ChatOpenAI(
    api_key=GPT_API_KEY, 
    model="gpt-4o-mini", 
    temperature=0
)

# Configure the model for structured output
structured_co = co.with_structured_output(UserInput, include_raw=True, method="json_mode")
structured_co2 = co.with_structured_output(UserInput)

try:
    prompt = "I want to perform analysis but don’t know how to do it"
    
    # Invoke the model
    AI_Response = structured_co.invoke(input=prompt)
    AI_Response_2 = structured_co2.invoke(input=prompt)

    # Parse the responses
    response = AI_Response['parsed'] 
    print("**1111111**")
    print(response)
    print("**22222**")
    print(AI_Response_2)

except Exception as e:
    print(f"An error occurred: {e}")

Description

Outputs:

Here are the outputs I received:

**1111111**
info_request='A situation where the user requests information, explanation, or a report (e.g., asking whether an analysis can be performed).' analysis_request='A situation where the user requests data processing, analysis, or visualization (e.g., requesting to perform operations on existing data/files).' other='Other content that does not fit into the above categories (e.g., a file upload request, general conversations).'
**22222**
info_request='I want to perform analysis but don’t know how to do it' analysis_request='' other=''

---

System Info

Problem:

I want to achieve the output in the second format (structured output matching the user input), but when I use the second method, I am unable to access input_tokens or token usage details.

---

Request:

Can you provide guidance or an example that demonstrates how to:

1. Get structured outputs in the desired format (like the second example).
2. Access token usage information such as input_tokens at the same time?

Thank you!

[t-n-y]

Hi,

i am also interested by knowing how to get get token usage with structured outputs using json schema. i can't find a clean way to do it neither

[ccurme]

In your example, AI_Response["raw"].usage_metadata should contain the usage information.

[vinzenzweber]

I am using ChatBedrock with Claude Sonnet 3.5. The include_raw=True does the job in this case. This is the gist:

structured_llm = llm.with_structured_output(CustomPromptResponse, include_raw=True)
response = structured_llm.invoke(user_prompt)
msg: AIMessage = response["raw"]
response: CustomPromptResponse = response["parsed"]
token_usage = msg.usage_metadata
log.debug(f"token usage:", extra=token_usage)
recommendation = response.feedback + "\n" + response.recommendation

And usage_metadata is a dict like {'input_tokens': 4412, 'output_tokens': 182, 'total_tokens': 4594}.

I hope this helps.

[t-n-y]

Thank you @vinzenzweber . I wasn't using the include_raw, now it's working !