Add support for update benchmarks

[bkontur]

Problem

The polkadot-sdk repository has a short-benchmarks pipeline that helps verify the correctness of benchmarks. In polkadot-fellowsrepository, we currently lack a similar pipeline for benchmark verification. Additionally, there is no existing "bench bot" to automatically regenerate weight files for polkadot-fellows runtimes. To maintain the integrity of our benchmarks and ensure the accuracy of weight files, we need to address these issues.

Proposed solution 1 - add pipeline

This pipeline will be responsible for running benchmarks and verifying their correctness on every PR.

Proposed solution 1 - regeneration

Since there is no existing "bench bot" for automatic regeneration of weight files for runtimes, we can implement a manual process or script.

Unresolved questions

• How and where to regenerate weights for this repo?
• What binary version to use? Benchmark commands are inside polkadot or polkadot-parachain binary)
• When to regenerate weight? Manually? On every release? On master?

cc: @ggwpez

[bkchr]

• How and where to regenerate weights for this repo?

Anyone can do this using the standard hardware. It should be documented here prominently what the standard hardware is. Then people also verify that the posted weights are in the same ball park, aka correct.

• What binary version to use? Benchmark commands are inside polkadot or polkadot-parachain binary)

Use the latest version and add a comment into the pull request about the version to make this process reproducible.

• When to regenerate weight? Manually? On every release? On master?

Manually. Fellowship doesn't has any infrastructure and depends on contributors to run the benchmarks. The fellowship members approving the pull request are then required to reproduce the weights to ensure that they are correct.

[xlc]

We will need a tool to compare the weights. I am not going to review all the numbers and check if all of them are close enough.

I already mentioned this in another issue, we should have the benchmark output a csv or json file so that we can easily process those numbers.

[acatangiu]

Manually. Fellowship doesn't has any infrastructure and depends on contributors to run the benchmarks. The fellowship members approving the pull request are then required to reproduce the weights to ensure that they are correct.

Would it be interesting to explore if someone (some-company or some chain-collective) would sponsor some infrastructure for this?
That way weights generation could be automated and the results also trusted without manual rechecking (depending on the level of control & trust over the provided infra).

[bkchr]

We will need a tool to compare the weights. I am not going to review all the numbers and check if all of them are close enough.

@ggwpez already has a tool for this! He can share this. I was also not expecting for you to do this manually :D

That way weights generation could be automated and the results also trusted without manual rechecking (depending on the level of control & trust over the provided infra).

This is the same trust level as trusting the guy that opens the PR. But yeah, theoretically it would work. However, this increases then the maintenance burden etc. Even in Parity all this stuff isn't working smoothly all the time. Having just one script that you can run on a machine sounds fine for me right now.

[xlc]

most of the devs won’t have access to a standard hardware machine locally so we will also need some guides on setting up one cloud providers

[bkchr]

We have already this: https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot

[shawntabrizi]

@ggwpez already has a tool for this! He can share this. I was also not expecting for you to do this manually :D

I believe this is the tool: https://github.com/ggwpez/substrate-weight-compare

[xlc]

Nice. Now we just need to have everything documented in a single place.

[ggwpez]

I believe this is the tool: https://github.com/ggwpez/substrate-weight-compare

Yes indeed. For example to get a diff on the ref-time component:

subweight compare commits --path-pattern "*/**/weights/*.rs" --method asymptotic --ignore-errors 1183e6b06108bf053c08e26c4c50fc644f572c7a 5312070b39a7529828c6b764f9c4f3ced206cfc3 --threshold 30 --unit time

And for the PoV size there is --unit proof. Output in markdown format looks like this:

File	Extrinsic	Old	New	Change [%]
system-parachains/asset-hubs/asset-hub-kusama/src/weights/pallet_assets_local.rs	destroy_approvals	40.80ms	139.20ms	+241.15
relay/polkadot/src/weights/pallet_staking.rs	cancel_deferred_slash	5.89ms	7.80ms	+32.29
relay/kusama/src/weights/pallet_staking.rs	cancel_deferred_slash	5.92ms	7.77ms	+31.40
relay/kusama/src/weights/pallet_xcm.rs	migrate_and_notify_old_targets	921.58us	639.08us	-30.65
relay/kusama/src/weights/pallet_election_provider_multi_phase.rs	create_snapshot_internal	2.40ms	1.66ms	-30.66
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs	transfer_reserve_asset	906.43us	619.93us	-31.61
relay/kusama/src/weights/pallet_balances_nis_counterpart_balances.rs	force_set_balance_killing	400.80us	270.86us	-32.42
relay/kusama/src/weights/pallet_xcm.rs	force_unsubscribe_version_notify	847.29us	564.77us	-33.34
relay/polkadot/src/weights/pallet_xcm.rs	send	513.88us	330.72us	-35.64
relay/kusama/src/weights/pallet_xcm.rs	notify_current_targets	764.12us	482.12us	-36.91
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs	deposit_reserve_asset	756.32us	472.16us	-37.57
relay/kusama/src/weights/pallet_xcm.rs	send	612.26us	329.64us	-46.16
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs	initiate_teleport	883.42us	474.10us	-46.33
relay/kusama/src/weights/pallet_xcm_benchmarks_fungible.rs	receive_teleported_asset	274.06us	146.28us	-46.62

There is also a web version, but i need to re-setup the server and add the runtimes repo. But it is possible to self-host as described in the readme.

Currently we have no trivial way to re-generate the weights. Sure there is the wiki description for the reference hardware and then anyone can run it, but its not straight forward, since the chain specs need to be generated and the node compiled first.

[svyatonik]

@bkontur FWIW I'm using this branch to do benchmarks locally. So something like:

1. cargo build --release -p chain-spec-generator --features runtime-benchmark
2. ./target/release/chain-spec-generator bridge-hub-polkadot-local --raw >bridge-hub-polkadot-local-raw.json
3. ../polkadot-sdk/target/release/polkadot-parachain-benchmarks benchmark pallet --chain bh-polkadot-local-raw.json --pallet pallet-bridge-grandpa --extrinsic "*" where the polkadot-parachain-benchmarks is the polkadot-parachain binary, compiled with --features runtime-benchmark

UPD: forgot to add thatyou need to use smth like bh-polkadot-local-raw.json for file name because of this. Think we need to change later