Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark: use default buffer sizes #5762

Merged
merged 1 commit into from
Nov 10, 2022

Conversation

atollena
Copy link
Collaborator

@atollena atollena commented Nov 2, 2022

The read and write buffer sizes are currently hardcoded to 128KB instead of the default 32KB. This can potentially lead to misleading benchmark results in the most common case, since most users presumably use the default value.

This change removes the custom value for read and write transport buffer sizes, both client and server side, so that benchmarks use the default values instead.

RELEASE NOTES: none

@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Nov 2, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: atollena / name: Antoine Tollenaere (9b8beba)

@atollena
Copy link
Collaborator Author

atollena commented Nov 2, 2022

The test failure seems unrelated to the change.

I'm running the benchmarks via go run benchmark/benchmain/main.go on my local machine (MacOS X/arm64) to compare the results, I'll post a diff of the results when this is done. Please let me know if you have further guidance on the environment to run the benchmarks on.

The read and write buffer sizes are currently hardcoded to 128KB instead of the default 32KB. This
can potentially lead to misleading benchmark results in the most common case, since most users
presumably use the default value.

This change removes the custom value for read and write transport buffer sizes, both client and
server side, so that benchmarks use the default values instead.
@atollena atollena force-pushed the benchmark-use-default-buffer-sizes branch from 9b8beba to 7e66658 Compare November 4, 2022 07:30
@easwars easwars added this to the 1.51 Release milestone Nov 4, 2022
@easwars
Copy link
Contributor

easwars commented Nov 5, 2022

@dfawley : For second set of eyes.

@atollena
Copy link
Collaborator Author

atollena commented Nov 7, 2022

I ran the benchmarks with the default values (resp/req size of 0, 1KB, 1MB - latency of 0ms - 40ms - max concurrency: 0, 8, 64, 512 - bandwidth 0 - 10Mbps). It's a lot of benchmarks - it takes 5h to run on an m5.2xl AWS instance, and results are difficult to analyse at least with the tools I currently have (I'm just diffing stdout).

Overall the results are as expected:

  • when payloads are very small, there is little performance impact: <10% throughput & latency in favour of larger buffer
  • the difference gets bigger with larger payloads - in benchmarks with 1MB payloads, with performance difference in the 50%. This is presumably due to the higher number of syscalls necessary to read/write the data.
  • The benchmark results seem consistent between unary, streaming & unconstraint, so I'm thinking of reducing the number of benchmarks by just testing one of them for future work on this.
  • The benchmarks with 40ms latency and limited throughput actually have better results with the default buffer size in some cases: unary-networkMode_none-bufConn_false-keepalive_false-benchTime_1s-trace_false-latency_40ms-kbps_10240-MTU_0-maxConcurrentCalls_512-reqSize_1024B-respSize_1024B-compressor_off-channelz_false-preloader_false is ~30% faster with smaller buffers. I can't explain why yet.

I don't think it's worth spending much more time looking at the benchmark results as part of this change. I plan to add a test dimension for the buffer sizes: #5757 to look into this further.

The results are here:

@zasweq zasweq modified the milestones: 1.51 Release, 1.52 Release Nov 8, 2022
@dfawley dfawley merged commit 457c2f5 into grpc:master Nov 10, 2022
jronak pushed a commit to jronak/grpc-go that referenced this pull request Nov 21, 2022
@atollena atollena deleted the benchmark-use-default-buffer-sizes branch November 25, 2022 08:20
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants