Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Salesforce xLAM model series #616

Merged
merged 5 commits into from
Sep 9, 2024
Merged

Conversation

zuxin666
Copy link
Contributor

@zuxin666 zuxin666 commented Aug 29, 2024

This PR add Salesforce/xLAM-7b-r, Salesforce/xLAM-8x7b-r, Salesforce/xLAM-8x22b-r models.

Note that the Salesforce/xLAM-8x7b-r model requires 8x40GB GPUs like A100 for inference, while the Salesforce/xLAM-8x22b-r model requires 8x80GB GPUs to run.

Tested these models with 8 H100 GPUs locally. The reference combined performance is as follows:

Here's the CSV table converted to markdown format:

Here's the CSV table converted to markdown format:

Rank Overall Acc Model AST Summary Exec Summary Irrelevance Detection Relevance Detection
1 87.31% xLAM-8x22b-r (FC) 82.76% 92.39% 74.96% 97.56%
2 83.38% xLAM-8x7b-r (FC) 78.18% 89.02% 72.35% 92.68%
3 80.33% xLAM-7b-r (FC) 74.00% 85.43% 72.88% 92.68%
4 80.18% xLAM-7b-fc-r (FC) 72.78% 87.68% 79.54% 80.49%
5 75.43% xLAM-1b-fc-r (FC) 65.72% 83.30% 60.65% 97.56%

@HuanzhiMao @CharlieJCJ Would you please take a look? Thanks for the great benchmark!

@HuanzhiMao
Copy link
Collaborator

Thanks for the PR @zuxin666!

One clarification question: Is Salesforce/xLAM-7b-r an FC model or a prompting model?

The rest looks good to me.

@HuanzhiMao HuanzhiMao added the BFCL-New Model Add New Model to BFCL label Aug 29, 2024
@liuzuxin
Copy link

Hi @HuanzhiMao , they are all FC models that share the same prompting format, so they can also share the same xLAMHandler. Thanks!

Copy link
Collaborator

@HuanzhiMao HuanzhiMao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShishirPatil ShishirPatil merged commit 9dec192 into ShishirPatil:main Sep 9, 2024
ShishirPatil pushed a commit that referenced this pull request Sep 15, 2024
…627, #635, and #638. (#639)

This PR updates the leaderboard to reflect the change in score due to
the following PR merge:

1. #608
2. #600
3. #616 
4. #623
5. #626
6. #627
7. #635 
8. #638

---------

Co-authored-by: Charlie Cheng-Jie Ji <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFCL-New Model Add New Model to BFCL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants