ChatCompletionCreateResponse incorrectly deserializes multiple tool calls when streaming

[skyegallup]

Describe the bug
Our application uses completion request streaming alongside OpenAI's recent support for multiple parallel tool calls. However, we have found that while OpenAI correctly returns multiple tool call objects in the stream, ChatCompletionCreateResponse always batches them into a single call with multiple function argument objects. This causes the first tool call's arguments to be malformed, and ignores all other tool calls from the API.

Your code piece

var result = new OpenAIResponse();
using var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(90));
await foreach (var completion in completionResult.WithCancellation(cancellationTokenSource.Token))
{
    if (completion.Successful)
    {
        var choice = completion.Choices.First();
        var tools = choice.Message.ToolCalls;
        Console.WriteLine(tools.Count.ToString());
        Console.WriteLine(tools.First().FunctionCall.Arguments.ToString());
    }
}

// Our prompt includes multiple tools, including "googleSearch" and "getURL".
// Our message to the agent: "Please search Google for cats and download the contents of www.wired.com."

Result
The code returns a single tool call containg the arguments of both tool calls: "{\"SearchTerm\": \"cats\"}{\"URL\": \"www.wired.com\"}"

Expected behavior
The code should return two separate tool calls in tools, each with its own arguments.

Desktop (please complete the following information):

• OS: Windows Server 2019
• Language: C#
• Version: v7.4.6

Additional context
Looking at a proxy log of the response from OpenAI, we can see that the API properly returns two separate tool call objects:

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"role":"assistant","content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_c4yp30JbgLn1lwxhAWjCCShC","type":"function","function":{"name":"googleSearch","arguments":""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"Se"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"archT"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"erm\": "}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"cat"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"s\"}"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"id":"call_iU0YaC5UfziyZN4bTQEKpsGS","type":"function","function":{"name":"getURL","arguments":""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"{\"UR"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"L\": \""}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"www.wi"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"red."}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{"tool_calls":[{"index":1,"function":{"arguments":"com\"}"}}]},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-8tjMROEisBcC2XbXNdjGD89ydAywJ","object":"chat.completion.chunk","created":1708293099,"model":"gpt-4-0125-preview","system_fingerprint":"fp_f084bcfc79","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}]}

data: [DONE]

This is what leads me to believe that this is an issue specifically with this library. I feel that this may be caused by the changes in #463, but I'm not familiar enough with the codebase to verify that.

[oferavnery]

I'm experiencing similar issues

[David-Buyer]

Hello,
it actually seems there's an use-case is not handled in changes i did, related to parallel tool calls responses in chat streaming completion mode API. I already have detected where the issue is and i will fix it asap ( this evening or at last tomorrow ).

Thank you for your details.