Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BFCL] Multi Turn Dataset and Possible Answer Fix (Base Category) #719

Merged
merged 4 commits into from
Oct 25, 2024
Merged

[BFCL] Multi Turn Dataset and Possible Answer Fix (Base Category) #719

merged 4 commits into from
Oct 25, 2024

Conversation

HuanzhiMao
Copy link
Collaborator

@HuanzhiMao HuanzhiMao commented Oct 24, 2024 •
edited
Loading

This PR addresses a few issues with the base multi turn entries:

  1. Some initial config values are in the wrong format, and would result in execution error even when running the ground truth function calls in order.
  2. Some entries in the initial config are not used by the _load_scenario method.
  3. Some ground truth function calls have wrong parameter values, and would result in {"error": "xxx"} when executed.
  4. Some functions are unreasonable in a real-life setting with its parameter or its functionality.

After this PR, all info in the initial config are used, and the ground truth function calls will not results in any error when executed in order.

This is part of the effort to thoroughly bug fix the multi turn categories. We will have more PR coming in the next few days.

Co-authored-by: Charlie Cheng-Jie Ji [email protected]
Co-authored-by: Fanjia-Yan [email protected]
Co-authored-by: VishnuSuresh27 [email protected]

@HuanzhiMao HuanzhiMao added the BFCL-Dataset BFCL Dataset-Related Issue label Oct 24, 2024
CharlieJCJ
CharlieJCJ approved these changes Oct 25, 2024
View reviewed changes
Copy link
Collaborator

@CharlieJCJ CharlieJCJ left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested changes in Message API.

LGTM, clear from dataset validation tests, ran bfcl v3 end-to-end.

@ShishirPatil ShishirPatil merged commit 2101b11 into ShishirPatil:main Oct 25, 2024
@HuanzhiMao HuanzhiMao mentioned this pull request Oct 25, 2024
Merged
ShishirPatil pushed a commit that referenced this pull request Oct 29, 2024
@HuanzhiMao @amitojsingh2022
[BFCL] Multi Turn Dataset Fix (Function Doc) (#722)
4c16dbb 
For functions used in the multi turn categories, we use automated
scripts to deterministically extract information from the function doc
string and compile the corresponding function docs. This PR addresses a
few issues with that pipeline.

1. [Fix] The doc strings are not in a uniform format, making info
extraction hard; they have been standardized.
2. [Fix] The compilation script is bugging and would sometimes output
wrong function doc with missing information.
3. [Fix] We have also add the compilation script to the `utils` folder,
as it may be useful to the community as well.

Following #719 , this is also part of the effort to thoroughly bug fix
the multi turn categories. We will have more PR coming in the next few
days.

---------

Co-authored-by: Amitoj Singh <[email protected]>
Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>

---------

Co-authored-by: amitojsingh2022 <[email protected]>
This was referenced Oct 29, 2024
Merged
Merged
Merged
ShishirPatil pushed a commit that referenced this pull request Oct 30, 2024
@HuanzhiMao
[BFCL] Multi Turn Dataset Fix (Base Category) (#723)
a79d891 
This PR fixes the ambiguous prompt issue and some wrong ground truth
issues for the multi_turn_base category. After this PR, the
multi_turn_base entries should be bug-free.

Following #719 and #722 , this is also part of the effort to thoroughly
bug fix the multi turn categories. We will have more PR coming in the
next few days.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
@HuanzhiMao HuanzhiMao mentioned this pull request Oct 31, 2024
Merged
HuanzhiMao added a commit that referenced this pull request Oct 31, 2024
@HuanzhiMao
[BFCL] Multi Turn Dataset Fix (Miss Func & Long Context) (#728)
12935b0 
This PR updates the question and ground truth for the
`multi_turn_miss_func` and `multi_turn_long_context` accordingly, since
they are augmented from `multi_turn_base` and the fix for the base
entries was finalized in #723.

Following #719, #722, #723 and #725, this is also part of the effort to
thoroughly bug fix the multi turn categories. There will be one more PR
coming for the `multi_turn_miss_param` category fix.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
HuanzhiMao added a commit that referenced this pull request Oct 31, 2024
@HuanzhiMao
[BFCL] Multi Turn Dataset Fix (Miss Param) (#732)
a471a3c 
This PR updates the question and ground truth for the
`multi_turn_miss_param` category, since they are augmented from
`multi_turn_base` and the fix for the base entries was finalized in
#723.

Following #719, #722, #723, #725 and #728, this is also part of the
effort to thoroughly bug fix the multi turn categories.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
HuanzhiMao added a commit that referenced this pull request Oct 31, 2024
@HuanzhiMao
[BFCL] Update Eval Metric for Multi Turn Irrelevance Scenarios (#725)
a84c1f7 
In the current metric, for the `multi_turn_miss_func` and
`multi_turn_miss_param` categories, the model is expected to output no
function calls when a turn is missing necessary information (either a
relevant function or parameter). This mirrors the standard for
irrelevance detection in single-turn scenarios. However, multi-turn
interactions introduce additional complexity.

For instance, if the user’s request is "go to the ABC folder and display
content of the XYZ file" but the `cd` function isn’t provided, the model
might reasonably attempt exploratory actions (like calling `pwd` or `ls`
to check its current location) before recognizing that it cannot
complete the task as requested. Ultimately, the model should recognize
that the user’s task is unachievable given the context.

To address this, we've updated the metric: a dummy function,
`flag_task_unachievable`, will now be provided for every multi-turn
entry. If the model determines that one or more tasks are unachievable,
it should explicitly invoke this function. During evaluation, any entry
where the model calls this function will be marked as correct for
irrelevance detection, even if other functions were called beforehand.

In addition, this PR addresses #664. The execution result for each turn
(for both the model and the ground truth) is also included as part of
the score output files to help with debugging.

Following #719, #722 and #723, this is also part of the effort to
thoroughly bug fix the multi turn categories. We will have more PR
coming in the next few days.
@HuanzhiMao HuanzhiMao mentioned this pull request Nov 9, 2024
Merged
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @VishnuSuresh27
[BFCL] Multi Turn Dataset and Possible Answer Fix (Base Category) (Sh…
e7e0fdd 
…ishirPatil#719)

This PR addresses a few issues with the base multi turn entries:

1. Some initial config values are in the wrong format, and would result
in execution error even when running the ground truth function calls in
order.
2. Some entries in the initial config are not used by the
`_load_scenario` method.
3. Some ground truth function calls have wrong parameter values, and
would result in `{"error": "xxx"}` when executed.
4. Some functions are unreasonable in a real-life setting with its
parameter or its functionality.

After this PR, all info in the initial config are used, and the ground
truth function calls will not results in any error when executed in
order.


This is part of the effort to thoroughly bug fix the multi turn
categories. We will have more PR coming in the next few days.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @amitojsingh2022 @VishnuSuresh27
[BFCL] Multi Turn Dataset Fix (Function Doc) (ShishirPatil#722)
e8c121a 
For functions used in the multi turn categories, we use automated
scripts to deterministically extract information from the function doc
string and compile the corresponding function docs. This PR addresses a
few issues with that pipeline.

1. [Fix] The doc strings are not in a uniform format, making info
extraction hard; they have been standardized.
2. [Fix] The compilation script is bugging and would sometimes output
wrong function doc with missing information.
3. [Fix] We have also add the compilation script to the `utils` folder,
as it may be useful to the community as well.

Following ShishirPatil#719 , this is also part of the effort to thoroughly bug fix
the multi turn categories. We will have more PR coming in the next few
days.

---------

Co-authored-by: Amitoj Singh <[email protected]>
Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>

---------

Co-authored-by: amitojsingh2022 <[email protected]>
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @VishnuSuresh27
[BFCL] Multi Turn Dataset Fix (Base Category) (ShishirPatil#723)
1c86a05 
This PR fixes the ambiguous prompt issue and some wrong ground truth
issues for the multi_turn_base category. After this PR, the
multi_turn_base entries should be bug-free.

Following ShishirPatil#719 and ShishirPatil#722 , this is also part of the effort to thoroughly
bug fix the multi turn categories. We will have more PR coming in the
next few days.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @VishnuSuresh27
[BFCL] Multi Turn Dataset Fix (Miss Func & Long Context) (ShishirPati…
966c0ac 
…l#728)

This PR updates the question and ground truth for the
`multi_turn_miss_func` and `multi_turn_long_context` accordingly, since
they are augmented from `multi_turn_base` and the fix for the base
entries was finalized in ShishirPatil#723.

Following ShishirPatil#719, ShishirPatil#722, ShishirPatil#723 and ShishirPatil#725, this is also part of the effort to
thoroughly bug fix the multi turn categories. There will be one more PR
coming for the `multi_turn_miss_param` category fix.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @VishnuSuresh27
[BFCL] Multi Turn Dataset Fix (Miss Param) (ShishirPatil#732)
0a7d039 
This PR updates the question and ground truth for the
`multi_turn_miss_param` category, since they are augmented from
`multi_turn_base` and the fix for the base entries was finalized in
ShishirPatil#723.

Following ShishirPatil#719, ShishirPatil#722, ShishirPatil#723, ShishirPatil#725 and ShishirPatil#728, this is also part of the
effort to thoroughly bug fix the multi turn categories.

---------

Co-authored-by: Charlie Cheng-Jie Ji
<[email protected]>
Co-authored-by: Fanjia-Yan
<[email protected]>
Co-authored-by: VishnuSuresh27
<[email protected]>
VishnuSuresh27 pushed a commit to VishnuSuresh27/gorilla that referenced this pull request Nov 11, 2024
@HuanzhiMao @VishnuSuresh27
[BFCL] Update Eval Metric for Multi Turn Irrelevance Scenarios (Shish…
d5cc8c8 
…irPatil#725)

In the current metric, for the `multi_turn_miss_func` and
`multi_turn_miss_param` categories, the model is expected to output no
function calls when a turn is missing necessary information (either a
relevant function or parameter). This mirrors the standard for
irrelevance detection in single-turn scenarios. However, multi-turn
interactions introduce additional complexity.

For instance, if the user’s request is "go to the ABC folder and display
content of the XYZ file" but the `cd` function isn’t provided, the model
might reasonably attempt exploratory actions (like calling `pwd` or `ls`
to check its current location) before recognizing that it cannot
complete the task as requested. Ultimately, the model should recognize
that the user’s task is unachievable given the context.

To address this, we've updated the metric: a dummy function,
`flag_task_unachievable`, will now be provided for every multi-turn
entry. If the model determines that one or more tasks are unachievable,
it should explicitly invoke this function. During evaluation, any entry
where the model calls this function will be marked as correct for
irrelevance detection, even if other functions were called beforehand.

In addition, this PR addresses ShishirPatil#664. The execution result for each turn
(for both the model and the ground truth) is also included as part of
the score output files to help with debugging.

Following ShishirPatil#719, ShishirPatil#722 and ShishirPatil#723, this is also part of the effort to
thoroughly bug fix the multi turn categories. We will have more PR
coming in the next few days.
HuanzhiMao added a commit that referenced this pull request Nov 19, 2024
@HuanzhiMao
[BFCL] Leaderboard Update, 11/17/2024 (#748)
2377862 
This PR updates the leaderboard to reflect the change in score due to
the following PR merge:

1. #719
2. #722
3. #723
4. #728 
5. #732
6. #725
7. #712
8. #733
9. #720 
10. #760 
11. #761 
12. #767
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CharlieJCJ CharlieJCJ CharlieJCJ approved these changes

Assignees
No one assigned
Labels
BFCL-Dataset BFCL Dataset-Related Issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@HuanzhiMao @CharlieJCJ @ShishirPatil