Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models #542

Merged
merged 4 commits into from
Jul 25, 2024
Merged

Add NousResearch/{Hermes-2-Pro-Llama-3-8B,Hermes-2-Theta-Llama-3-8B} models #542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2e2e0e5
Add NousResearch/Hermes-2-Pro-Llama-3-8B model
alonsosilvaallende Jul 22, 2024
82a36a7
Add NousResearch/Hermes-2-Theta-Llama-3-8B model
alonsosilvaallende Jul 23, 2024
0b65e2f
Merge branch 'main' into main
HuanzhiMao Jul 24, 2024
e3ba734
Add latency info
HuanzhiMao Jul 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion berkeley-function-call-leaderboard/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Below is *a table of models we support* to run our leaderboard evaluation agains
|mistral-small-2402 | Prompt|
|mistral-tiny-2312 | Prompt|
|Nexusflow-Raven-v2 | Function Calling|
|NousResearch/Hermes-2-Pro-Mistral-7B 💻| Function Calling|
|NousResearch/Hermes-2-{Pro-Llama-3-8B,Pro-Mistral-7B,Theta-Llama-3-8B} 💻| Function Calling|
|snowflake/arctic | Prompt|
|nvidia/nemotron-4-340b-instruct| Prompt|
|THUDM/glm-4-9b-chat 💻| Function Calling|
Expand Down
14 changes: 14 additions & 0 deletions berkeley-function-call-leaderboard/eval_checker/eval_runner_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,12 +318,24 @@
"Databricks",
"Databricks Open Model",
],
"NousResearch/Hermes-2-Pro-Llama-3-8B": [
"Hermes-2-Pro-Llama-3-8B (FC)",
"https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B",
"NousResearch",
"apache-2.0",
],
"NousResearch/Hermes-2-Pro-Mistral-7B": [
"Hermes-2-Pro-Mistral-7B (FC)",
"https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B",
"NousResearch",
"apache-2.0",
],
"NousResearch/Hermes-2-Theta-Llama-3-8B": [
"Hermes-2-Theta-Llama-3-8B (FC)",
"https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B",
"NousResearch",
"apache-2.0",
],
"meta-llama/Meta-Llama-3-8B-Instruct": [
"Meta-Llama-3-8B-Instruct (Prompt)",
"https://llama.meta.com/llama3",
Expand Down Expand Up @@ -475,6 +487,8 @@
"deepseek-ai/deepseek-coder-6.7b-instruct": 909,
"google/gemma-7b-it": 95,
"NousResearch/Hermes-2-Pro-Mistral-7B": 135,
"NousResearch/Hermes-2-Pro-Llama-3-8B": 77,
"NousResearch/Hermes-2-Theta-Llama-3-8B": 73,
"meta-llama/Meta-Llama-3-8B-Instruct": 73,
"meta-llama/Meta-Llama-3-70B-Instruct": 307,
"gorilla-openfunctions-v2": 83,
Expand Down
2 changes: 2 additions & 0 deletions berkeley-function-call-leaderboard/model_handler/constant.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,9 @@
"meetkai/functionary-medium-v2.2-FC",
"meetkai/functionary-small-v2.4-FC",
"meetkai/functionary-medium-v2.4-FC",
"NousResearch/Hermes-2-Pro-Llama-3-8B",
"NousResearch/Hermes-2-Pro-Mistral-7B",
"NousResearch/Hermes-2-Theta-Llama-3-8B",
"command-r-plus-FC",
"command-r-plus-FC-optimized",
"THUDM/glm-4-9b-chat",
Expand Down
2 changes: 2 additions & 0 deletions berkeley-function-call-leaderboard/model_handler/handler_map.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@
"meetkai/functionary-small-v2.4-FC": FunctionaryHandler,
"meetkai/functionary-medium-v2.4-FC": FunctionaryHandler,
"databricks-dbrx-instruct": DatabricksHandler,
"NousResearch/Hermes-2-Pro-Llama-3-8B": HermesHandler,
"NousResearch/Hermes-2-Pro-Mistral-7B": HermesHandler,
"NousResearch/Hermes-2-Theta-Llama-3-8B": HermesHandler,
"meta-llama/Meta-Llama-3-8B-Instruct": LlamaHandler,
"meta-llama/Meta-Llama-3-70B-Instruct": LlamaHandler,
"command-r-plus-FC": CohereHandler,
Expand Down
6 changes: 3 additions & 3 deletions berkeley-function-call-leaderboard/model_handler/hermes_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ def _format_prompt(prompt, function, test_category):
)
pydantic_format = """{"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}"""
tool_call_format = """{"arguments": <args-dict>, "name": <function-name>}"""
formatted_prompt = """
<|im_start|>system
formatted_prompt = """<|im_start|>system
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools>{function}</tools> Use the following pydantic model json schema for each tool call you will make: {pydantic_format} For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{tool_call_format}
Expand All @@ -26,7 +25,8 @@ def _format_prompt(prompt, function, test_category):
<|im_start|>user
{prompt}
<|im_end|>
"""
<|im_start|>assistant
"""
return formatted_prompt.format(
function=function,
pydantic_format=pydantic_format,
Expand Down