[BFCL] Fix language_specific_pre_processing for Java and JavaScript Test Category #538
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HuanzhiMao
added a commit
to HuanzhiMao/gorilla
that referenced
this pull request
Jul 22, 2024
Squashed commit of the following: commit 407d443 Author: Huanzhi Mao <[email protected]> Date: Sun Jul 21 00:35:32 2024 -0700 Add missing description field for 2 entry in javascript dataset commit 2c0d311 Author: Huanzhi Mao <[email protected]> Date: Sat Jul 20 23:37:20 2024 -0700 update change log commit 0d3bc83 Author: Huanzhi Mao <[email protected]> Date: Sat Jul 20 23:03:43 2024 -0700 clean up commit 33b9fc3 Author: Huanzhi Mao <[email protected]> Date: Sat Jul 20 22:44:05 2024 -0700 remove unused parameter string_param from all model_handlers commit ad7c337 Author: Huanzhi Mao <[email protected]> Date: Sat Jul 20 22:43:27 2024 -0700 fix language_specific_pre_processing
ShishirPatil
approved these changes
Jul 22, 2024
ShishirPatil
pushed a commit
that referenced
this pull request
Jul 25, 2024
…ry, Part 2 (#545) This PR fixes the function doc pre-processing issues for the Java and JavaScript test categories, following up on PR #538. - Some unnecessary steps in the `convert_to_tool` function are removed. These steps should not exist as not every model handler calls the `convert_to_tool` function (for example, the OSS models) and would unfairly benefit the models that use it. To make sure that every model gets the same pre-processed function doc, the pre-processing phase needs to be in the `language_specific_pre_processing` function (which is used by every handler). - Properly handle the inner element type for nested types.
ShishirPatil
pushed a commit
that referenced
this pull request
Jul 28, 2024
…551) This PR updates the leaderboard to reflect the changes in score due to the following PR merge. - #538 - #545 The scores for the following categories are affected: - Java - JavaScript Consequently, the aggregated score also gets updated: - Simple Func AST - AST Summary - Overall Acc      
aw632
pushed a commit
to vinaybagade/gorilla
that referenced
this pull request
Aug 22, 2024
…est Category (ShishirPatil#538) In our BFCL official communication channels, including the evaluation manual blog, GitHub issue replies (such as ShishirPatil#424), and our Discord channel, we have previously stated the following: > For Java and JavaScript test category, before querying the model, we do some pre-processing on the prompt and function document. Specifically, at the end of the prompt, we will explicitly state that `the provided function is in Java 8/JavaScript/Python syntax`. And for parameter types that are not native to JSON, we will change their type to `String` (since `String` is JSON compatible) and add in the parameter description that `f" This is Java/JavaScript {value['type']} in string representation."` > As an example, when expecting type `ArrayList`, model will get the instruction that this is a `String` type parameter with the parameter description containing `"This is Java ArrayList in string representation."`, and thus the model should output the value in `String` format (eg, `"new ArrayList<>(Arrays.asList(10, 20, 30))"`), which is JSON compatible. However, the code for `language_specific_pre_processing` did not implement this correctly. Due to an indentation issue, the parameter description was only modified when the parameter type was `any`, and the part where the parameter type is cast to `String` was never implemented. This issue was unnoticed until PR ShishirPatil#516 was merged because of the double-casting problem. It *significantly impacts* the evaluation score for the Java and JavaScript categories. We will update the leaderboard very soon. This PR: - Addresses the above issue and ensures that the evaluation logic aligns with the previously described behaviour - Updates two entries in the JavaScript dataset, due to their parameters missing a `description` field - Index: `14, 45` We sincerely apologize for the oversight.
aw632
pushed a commit
to vinaybagade/gorilla
that referenced
this pull request
Aug 22, 2024
…ry, Part 2 (ShishirPatil#545) This PR fixes the function doc pre-processing issues for the Java and JavaScript test categories, following up on PR ShishirPatil#538. - Some unnecessary steps in the `convert_to_tool` function are removed. These steps should not exist as not every model handler calls the `convert_to_tool` function (for example, the OSS models) and would unfairly benefit the models that use it. To make sure that every model gets the same pre-processed function doc, the pre-processing phase needs to be in the `language_specific_pre_processing` function (which is used by every handler). - Properly handle the inner element type for nested types.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In our BFCL official communication channels, including the evaluation manual blog, GitHub issue replies (such as #424), and our Discord channel, we have previously stated the following:
However, the code for
language_specific_pre_processingdid not implement this correctly. Due to an indentation issue, the parameter description was only modified when the parameter type wasany, and the part where the parameter type is cast toStringwas never implemented.This issue was unnoticed until PR #516 was merged because of the double-casting problem.
It significantly impacts the evaluation score for the Java and JavaScript categories. We will update the leaderboard very soon.
This PR:
descriptionfield14, 45We sincerely apologize for the oversight.