Add NousResearch Hermes models to no cost models list

[alonsosilvaallende]

Add NousResearch Hermes models to no cost models list

[HuanzhiMao]

Hi @alonsosilvaallende,
The Hermes model should not be added to the NO_COST_MODELS list.
We use the following formula to estimate the cost and latency for locally-hosted models.
Since the Hermes model can be loaded into 8 x V100 machines, it will have the Cost and Latency fields computed as above.

[alonsosilvaallende]

Thank you very much for your explanation @HuanzhiMao
What does 'no cost' mean? I see that Salesforce/xLAM-7b-fc-r has no cost. Does that mean the cost could not be determined?

[HuanzhiMao]

Thank you very much for your explanation @HuanzhiMao What does 'no cost' mean? I see that Salesforce/xLAM-7b-fc-r has no cost. Does that mean the cost could not be determined?

Yes, because the xLAM model cannot be loaded into 8 x V100 machines (bfloat16 is not compatible with V100). See here.

[alonsosilvaallende]

I understand. The name was misleading me. Last question: Also the functionary small models meetkai/functionary-small-v2.4 cannot be run on 8 x V100 machines?

[HuanzhiMao]

I understand. The name was misleading me. Last question: Also the functionary small models meetkai/functionary-small-v2.4 cannot be run on 8 x V100 machines?

Yes, you are right. functionary-small does require bfloat16. We will fix it.