[BFCL] The generation in v3 is too slow

[XuHwang]

Describe the issue
A clear and concise description of what the issue is.

ID datapoint

1. Datapoint / Model Handler permalink:
2. Issue:
3. Gorilla repo commit #:

What is the issue

After migration to version 3, where the vllm API serving strategy is adopted, the generation speed shows a severe degradation. It takes about 6x more time to get all results.

Proposed Changes

{
'previous_datapoint':[],
'updated_datapoint':[]
}

Additional context
Add any other context about the problem here.

[ShishirPatil]

Hey @XuHwang Which models are you trying it on, and what's your GPU config? I don't know about the 6X, but yeah we do expect it to take significantly longer since a) multi-turn responses, and b) long-context responses both increase latency.

[XuHwang]

Hey @XuHwang Which models are you trying it on, and what's your GPU config? I don't know about the 6X, but yeah we do expect it to take significantly longer since a) multi-turn responses, and b) long-context responses both increase latency.

Hey, thanks for the reply.

I evaluate my own model with oss_handler on 4*v100-32G. And the model size is 8B. The generation process would cost about 10 min in version 2 while it takes more than 1 hour in version 3. I wonder whether it is caused by the gap between the vllm API serving strategy and batch_generate (such as the API serving strategy would handle one sample each time while the batch_generate would handler more samples?)

[HuanzhiMao]

Hey @XuHwang,
Could you try #671 and see if it solves your issue?