[BFCL] Add support for Writer models and Palmyra X 004

[samjulien]

This PR adds support for Writer models and our latest Palmyra X 004 to BFCL. Thank you!

[HuanzhiMao]

Thanks for the PR @samjulien !

I noticed that here, you intentionally removed all the response field from the function doc.
For model that doesn't support response field, we add those information to the end of the function description because we don't want information loss (code here). Is there a reason why you chose to do it that way?

[samjulien]

Thanks for the PR @samjulien !

I noticed that here, you intentionally removed all the response field from the function doc. For model that doesn't support response field, we add those information to the end of the function description because we don't want information loss (code here). Is there a reason why you chose to do it that way?

Ah thank you! I’ll add a check for our models there and remove from our handler.

[samjulien]

Hi @HuanzhiMao, I made that change you suggested and rebased the branch to be updated with main (note: force push), but I'm getting some weird errors now that I wasn't getting before these updates. Looks like it is happening in multi_threaded_inference, getting KeyError: 'function'. I'll keep investigating, but maybe you can take a look/run it locally? Or have you seen this with other providers recently? Anything to go on would be helpful. Thanks!

[HuanzhiMao]

Hey @samjulien, I have submitted a PR to your fork. samjulien#1

Two things I changed:

1. All the methods ending with _prompting are removed, because they are never used. Palmyra is a Funciton Calling model, and so only those _FC methods will be used.
2. I made the WriterHandler inherit from the OpenAIHandler since they share the exact same logic for decoding, tool handling, etc. The only difference is _query_FC because writerai use self.client.chat.chat instead of self.client.chat.completions.create

I have tested the code locally and it is working perfectly fine.

[samjulien]

Thank you so much @HuanzhiMao, great thinking! I really appreciate you doing that. I've merged it in.

One other question: We don't usually append -FC to palmyra-x-004 since we don't have a separate non-FC model, but it seemed like it is necessary for the tests to run correctly. Am I right on that?

[HuanzhiMao]

Thank you so much @HuanzhiMao, great thinking! I really appreciate you doing that. I've merged it in.

One other question: We don't usually append -FC to palmyra-x-004 since we don't have a separate non-FC model, but it seemed like it is necessary for the tests to run correctly. Am I right on that?

Yes. The -FC is mainly to signal the BFCL user that this is a function calling model (instead of a prompt model). You can see that when we call the endpoint, we remove the -FC in the model name. So under the hood, we are using the palmyra-x-004 model for inference:

api_response = self.client.chat.chat(
messages=message,
model=self.model_name.replace("-FC", ""),
temperature=self.temperature,
tools=tools,
tool_choice="auto",
)

If you don't want the _FC in model name, we can also remove it. Either way is fine.

[HuanzhiMao]

LGTM. Thanks for the PR!

[HuanzhiMao]

Update: I removed the _FC in the model name.

[samjulien]

Thank you so much @HuanzhiMao! Let me know if you need an API key for the leaderboard.

[HuanzhiMao]

Thank you so much @HuanzhiMao! Let me know if you need an API key for the leaderboard.

DM-ed via Discord.

[Gdewilde]

Nice addition. Why is this model not visible on the leaderboard?