Block stream benchmark tests

[iamalwaysuncomfortable]

Ready for Review!

Motivation

The block streaming components should be benchmarked individually under stress that is normally expected to be seen in production. This PR introduces a bench script format + tools to simulate block streaming load.

In this PR

• Bench script to test a particular component, but meant to be generalizable to others
• Function to create mock ledger blocks with content similar to those seen in mainnet

Addresses #1707

Future Work

• Write benchmarks for each block streaming component
• Benchmark the entire integrated block streaming pipeline

[iamalwaysuncomfortable]

Basically, out of a review of this currently, I'd like inputs on the fundamental approach which is:

1. Building blocks that have content close to what mainnet blocks currently have and using those for benchmark tests
2. Benchmarking individual blocksteram components by driving the streams individually to benchmark the speed of the logic being executed when the stream are driven
3. Further I think the natural extension is combining components into maybe 1-2 different configurations of chained stream components and benchmarking integrated components in a similar way to the above.

Given those 3 points, does the team have any commentary on the approach, how it could be better, etc.?

@remoun , @jcape

[remoun]

Basically, out of a review of this currently, I'd like inputs on the fundamental approach which is:

1. Building blocks that have content close to what mainnet blocks currently have and using those for benchmark tests

SGTM, if it's easy enough to approximate the mainnet config(s).

2. Benchmarking individual blocksteram components by driving the streams individually to benchmark the speed of the logic being executed when the stream are driven

It seems higher leverage/more useful to benchmark the end-to-end pipelines.

3. Further I think the natural extension is combining components into maybe 1-2 different configurations of chained stream components and benchmarking integrated components in a similar way to the above.

Agreed. I've started defining helpers that assemble the pipelines we expect to support, e.g. consensus node, consensus client, gRPC fanout repeater.

[remoun]

Hmm, I'm not sure how or why this got closed.