[V1][Frontend] Coalesce bunched RequestOutputs

[njhill]

Under high load it's possible for the frontend per-request asyncio queues to back up, with the next token(s) arriving before existing ones are streamed back to the user.

In this case there's no reason for them to be emitted as separate outputs in subsequent iterations. By concatenating them into a single output it reduces the number of tasks / context switches / response messages to be handled and means those additional "ready" tokens should reach the user faster.

Benchmarking shows a small benefit to latencies, TTFT in particular.

On an A100:

VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.2-1B-Instruct --disable-log-requests --port 8001 --max-num-batched-tokens 2048 --no-enable-prefix-caching --uvicorn-log-level=error 

    python3 benchmark_serving.py \
        --model meta-llama/Llama-3.2-1B-Instruct \
        --dataset-name sharegpt \
        --dataset-path /workspace/ShareGPT_V3_unfiltered_cleaned_split.json \
        --port 8001 \
        --ignore-eos \
        --num-prompts 6000 \
        --request-rate 60 \
        --backend vllm

Before

============ Serving Benchmark Result ============
Successful requests:                     6000      
Benchmark duration (s):                  107.69    
Total input tokens:                      1322102   
Total generated tokens:                  1205048   
Request throughput (req/s):              55.72     
Output token throughput (tok/s):         11190.46  
Total Token throughput (tok/s):          23467.91  
---------------Time to First Token----------------
Mean TTFT (ms):                          56.86     
Median TTFT (ms):                        47.85     
P99 TTFT (ms):                           128.49    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.03     
Median TPOT (ms):                        8.55      
P99 TPOT (ms):                           17.88     
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.88      
Median ITL (ms):                         8.59      
P99 ITL (ms):                            24.48     
==================================================

After

============ Serving Benchmark Result ============
Successful requests:                     6000      
Benchmark duration (s):                  107.34    
Total input tokens:                      1322102   
Total generated tokens:                  1205048   
Request throughput (req/s):              55.90     
Output token throughput (tok/s):         11226.66  
Total Token throughput (tok/s):          23543.84  
---------------Time to First Token----------------
Mean TTFT (ms):                          51.97     
Median TTFT (ms):                        45.87     
P99 TTFT (ms):                           115.81    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.44      
Median TPOT (ms):                        8.41      
P99 TPOT (ms):                           15.95     
---------------Inter-token Latency----------------
Mean ITL (ms):                           9.52      
Median ITL (ms):                         8.28      
P99 ITL (ms):                            24.00     
==================================================

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @njhill.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[WoosukKwon]

@njhill What's the status of this PR? Would it be possible to merge this PR asap?

Also, could you provide a performance benchmark? In which cases would this PR help most?

[njhill]

@WoosukKwon it's ready but I am looking into unexpected CI test failure (unclear why this change would result in a CUDA OOM).

I will also run some more benchmarks.

[njhill]

Tests should now be fixed.

[mgoin]

Can be in a future PR, but we should check for invalid sampling params like n > 1 here

[njhill]

Yes I guess that's a general V1 thing, we should be explicitly rejecting requests that include unsupported parameters if we aren't already.