Create benchmark scripts for demo-rollup running in native mode

[dubbelosix]

Description

We want to be able to measure the native performance of the SDK, and to this purpose we're benchmarking demo-rollup running in native mode. the process for the benching is as follows

• Generate 100 blocks containing 1 blob each with 10k transfer transactions (defaults, but can be configured)
• Once the blocks are loaded into memory, we start the timers and measure how long the blocks take to be processed.
• The primary pieces of code we are measure are
  • begin_slot
  • apply_blob
  • end_slot

demo-rollup/benches/README.md contains the detailed documentation

Tools used:

• criterion
• manual timers
• prometheus metrics (metrics only. no http server, no scraping and no visualization tools for now)
• flamegraph
• make

There are 2 scripts for benchmarking -

• rollup_bench.rs - This uses criterion and pre-populates a "large" amount of data and lets the number of iterations be decided by criterion. we do constrain measurement time to 20 seconds to prevent too many iterations leading in an indexing error
• rollup_coarse_measure.rs
  • This uses coarse grained manual timers and outputs the timings as well the tps
  • Can also output the readings as a histogram for prometheus. the data is currently a json, but can be fed into visualization tools (this will be done in another issue)

Also contains a Makefile for wrapping all the scripts + cargo flamegraph (sample output shown)
Added README.md explaining the bench methodology

Linked Issues

• Related to # (356)

Approved by accident

[codecov]

Codecov Report

Merging #458 (6cf3076) into main (f76a52a) will decrease coverage by 0.1%.
The diff coverage is 100.0%.

Impacted Files	Coverage Δ
rollup-interface/src/state_machine/mocks.rs	61.9% <ø> (ø)
...ystem/sov-modules-stf-template/src/app_template.rs	82.6% <100.0%> (ø)

... and 1 file with indirect coverage changes

[citizen-stig]

@dubbelosix I am trying to run benchmark on my baremetal machine with 16 cores and it doesn't seem to fit in 20 seconds:

$ make criterion
Running criterion bench with 10000 transactions per block
Method: Criterion
Output: Criterion
    Updating crates.io index
....
    Finished bench [optimized] target(s) in 16m 08s
     Running benches/rollup_bench.rs (/home/stig/workspace/sovereign-sdk/target/release/deps/rollup_bench-988dda5198a92fac)
     Running benches/rollup_bench.rs (/home/stig/workspace/sovereign-sdk/target/release/deps/rollup_bench-988dda5198a92fac)
Benchmarking rollup main loop
Benchmarking rollup main loop: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 20.0s. You may wish to increase target time to 967.9s.
Benchmarking rollup main loop: Collecting 10 samples in estimated 967.87 s (10 iterations)

What can be a problem? htop show no CPU activity at all.

[dubbelosix]

@dubbelosix I am trying to run benchmark on my baremetal machine with 16 cores and it doesn't seem to fit in 20 seconds:

$ make criterion
Running criterion bench with 10000 transactions per block
Method: Criterion
Output: Criterion
    Updating crates.io index
....
    Finished bench [optimized] target(s) in 16m 08s
     Running benches/rollup_bench.rs (/home/stig/workspace/sovereign-sdk/target/release/deps/rollup_bench-988dda5198a92fac)
     Running benches/rollup_bench.rs (/home/stig/workspace/sovereign-sdk/target/release/deps/rollup_bench-988dda5198a92fac)
Benchmarking rollup main loop
Benchmarking rollup main loop: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 20.0s. You may wish to increase target time to 967.9s.
Benchmarking rollup main loop: Collecting 10 samples in estimated 967.87 s (10 iterations)

What can be a problem? htop show no CPU activity at all.

so I checked this issue and it seems like you really can't control the exact amount of time criterion runs - it only takes the measurement time as a hint. tried looking to see if there are other frameworks that have more fine grained control, but didn't find too much. that's one of the reasons i also have the other "rollup_coarse_measure.rs"