Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(runtime, ci): finish migration to bare-metal benchmark runner and update weights #3994

Merged
merged 6 commits into from
Jun 12, 2024

Conversation

ByteNacked
Copy link
Member

@ByteNacked ByteNacked commented Jun 3, 2024

What was done:

A migration of the benchmark runner was performed, moving from a virtual host to a bare-metal host to increase the reproducibility of benchmark results.

Since the bare-metal host differs somewhat in performance from the equivalent virtual host, we investigated these differences through several benchmark runs and calculated the average discrepancy between them. As a result, a 10% adjustment was introduced to the instruction weights, aligning the instruction weight results with those on the virtual host.

Additionally, for the multi-threaded benchmarks of the gear_builtin pallet, it was decided to fix the execution of these benchmarks to 4 CPU cores, again to improve the reproducibility of the results.

  • Added script which adjust Instruction weights by 10 % when running on bare metal runner
  • Run multi threaded benches (pallet_gear_builtin) on fixed number of cores (4 cores)
  • Update weights from bare-metal run

@reviewer-or-team

@ByteNacked ByteNacked self-assigned this Jun 3, 2024
@ByteNacked ByteNacked force-pushed the rmasl-bench-adjust-weights branch from 90ad5ed to 734aecb Compare June 3, 2024 21:12
@ByteNacked ByteNacked changed the title Update bench script for baremetal Update baremetal CI script Jun 3, 2024
@ByteNacked ByteNacked requested review from grishasobol and breathx June 3, 2024 21:23
@ByteNacked ByteNacked force-pushed the rmasl-bench-adjust-weights branch from c624131 to 7cd88a9 Compare June 4, 2024 08:09
@ByteNacked ByteNacked added the A0-pleasereview PR is ready to be reviewed by the team label Jun 4, 2024
@ByteNacked ByteNacked changed the title Update baremetal CI script Finish migration to bare-metal benchmark runner Jun 4, 2024
@ByteNacked ByteNacked force-pushed the rmasl-bench-adjust-weights branch from 4c2c58d to b0bddd1 Compare June 4, 2024 14:45
@ByteNacked
Copy link
Member Author

Should InstructionWeight version be incremented in this PR due to some instruction weight change?

@breathx
Copy link
Member

breathx commented Jun 11, 2024

Should InstructionWeight version be incremented in this PR due to some instruction weight change?

Unnecessary, it's all up to you, on release tagging I'll anyway double check all versions

@breathx
Copy link
Member

breathx commented Jun 11, 2024

Please resolve master brach conflicts

@breathx breathx added A2-mergeoncegreen PR is ready to merge after CI passes and removed A0-pleasereview PR is ready to be reviewed by the team labels Jun 11, 2024
@ByteNacked ByteNacked changed the title Finish migration to bare-metal benchmark runner feat(runtime, ci): finish migration to bare-metal benchmark runner and update weights Jun 11, 2024
@ByteNacked ByteNacked merged commit 76d11bd into master Jun 12, 2024
10 checks passed
@ByteNacked ByteNacked deleted the rmasl-bench-adjust-weights branch June 12, 2024 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A2-mergeoncegreen PR is ready to merge after CI passes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants