Offline LLM Engine Benchmark Throughput

[zolinthecow]

Motivation

#1865
Add throughput benchmark for engine.generate

Modifications

Added ability to specify an engine instead of an API url in the benchmarks

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[zolinthecow]

@ByronHsu do you want me to add this to CI?

[merrymercy]

This PR tries to reuse the script bench_serving.py. However, I think a better approach is to use a standalone script.

Reason:

• bench_serving.py is for online serving, but the most common use case of the Engine API is for offline use cases. We should benchmark the non-async version of the Engine to see what the maximum throughput we can get without all of the streaming/asyncio overhead
• We want to pass in many other arguments to the server. The new script should be similar to bench_latency.py, which takes in the full ServerArgs as arguments.

  sglang/python/sglang/bench_latency.py

  Lines 536 to 541 in 760552e

  	parser = argparse.ArgumentParser()
  	ServerArgs.add_cli_args(parser)
  	BenchArgs.add_cli_args(parser)
  	args = parser.parse_args()
  	server_args = ServerArgs.from_cli_args(args)
  	bench_args = BenchArgs.from_cli_args(args)

Can you try to write a standalone script bench_offline_throughput.py that takes the same arguments as bench_latency.py?

[zolinthecow]

will do

[zolinthecow]

@merrymercy updated script, what do you think

[merrymercy]

This looks better! Sorry for the back-and-forth, but I think we should support loading real datasets and support better random dataset generation, similar to the one in bench_serinvg.py.

The bench_latency.py uses a simple way to generate synthetic data because it does not support continuous batching or inputs with variable lengths. The engine supports everything so we can use more realistic data.

[merrymercy]

Also, please add a unit test here https://github.com/sgl-project/sglang/blob/main/test/srt/test_srt_engine.py to run this benchmark for 10 random prompts.

[zolinthecow]

@merrymercy made the changes

[ByronHsu]

Can we have an option for runtime backend? So we can easily do benchmark between runtime.generate and engine.generate. Related to #1872

[zolinthecow]

sure. i would suggest that we keep input + output since it's a pretty standard way to measure throughput (total system throughput) as far as i know. definitely will put both though. maybe add input + output to bench_serving as well?

[merrymercy]

The script now looks good. @ByronHsu can you verify the output format? @zolinthecow Can you resolve the remaining comments? Then we can merge this soon!

[merrymercy]

LGTM. @ByronHsu you can do the final merge.

[ByronHsu]

Slightly refactored the code and attempted to fix the test. Can merge if the test passes

[ByronHsu]

Thanks for the contribution!! @zolinthecow

[merrymercy]

@zolinthecow great work!