[FEATURE] Implement Dynamic SplitFuse

[casper-hansen]

Dear vLLM maintainers @WoosukKwon and @zhuohan123 (@Yard1),

DeepSpeed has released its serving framework which claims to be faster than vLLM. The main speedup comes from Dynamic SplitFuse which is a technique that does the following:

• Long prompts are decomposed into much smaller chunks and scheduled across multiple forward passes (iterations) with only the final pass performing any generation.

• Short prompts will be composed to exactly fill a target token budget. Even short prompts may be decomposed to ensure the budget is precisely met and the forward sizes are well-aligned.

Code: https://github.com/microsoft/DeepSpeed-MII
Background: https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-fastgen

Llama 13B (1x A100-80GB):

Llama 70B (4x A100x80GB with TP):

[irasin]

LGTM

[thesues]

Hi, is there any progress right now?

[shixianc]

Do we have an ETA? 😊

[tdene]

Hi @WoosukKwon @zhuohan123

The absence of a chunked prefill implementation in vllm is a major blocker. Any kind of timeline or regular communication on progress towards a chunked prefill implementation would be immensely helpful, just to allow for future planning.

[sh1ng]

Keeping a batch with aligned length definitely helps #2357

[njhill]

Looks like someone has started working on this: #3106

[hmellor]

Chunked prefill is now supported