Create benchmarks for queries

[facundomedica]

Summary

Add benchmarks that run queries against a pre-defined data directory in order to know wether a PR results in an improvement or deterioration in performance.

Problem Definition

Currently, in many cases we don't have any metrics to understand the impact of merging a PR.

Proposal

Create a series of benchmarks like this:

func BenchmarkQueryStakingDelegations(b *testing.B) {
  // create app and load state from a db directory

  for i := 0; i < b.N; i++ {
    // ...
  } 
}

Proposed by @alexanderbez

[alexanderbez]

Love this! This will give us a good way to measure the baseline and any changes or improvements. I propose we have benchmarks for the following queries:

• Balance (x/bank)
• AllBalances (x/bank)
• Proposals (x/gov)
• Votes (x/gov)
• SigningInfos (x/slashing)
• ValidatorDelegations (x/staking) <- this will be a good one

We can use https://quicksync.io/networks/cosmos.html to get data directories to feed into the app

[peterbourgon]

Just want to node that, in order for benchmarks like these to produce results that are reliably reproducible and comparable, it's important that the state they operate on is committed to version control. A good litmus test: git checkout <rev> followed by go test -bench=whatever should always do exactly the same thing for everyone, and shouldn't rely on some script to populate test data or whatever. Snapshots from quicksync.io are (of course) enormous, so if you want to use them as a starting point that's cool, just be sure to extract and commit a (tiny) representative sample of the data.

[alexanderbez]

Just want to node that, in order for benchmarks like these to produce results that are reliably reproducible and comparable, it's important that the state they operate on is committed to version control.

We're not going to commit the state to version control. The state we're benchmarking on can range from GBs to TBs. What we will do instead if post the results along with the profiles and reproduction steps (including hashes of state files).

However, we can include a script that automates all of this.

[peterbourgon]

Exactly my point — benchmarks are meaningful when they exercise a representative, well-defined, and ultimately small sample of a codepath, so that go bench can run it over and over for the stats. Loading up GBs of data adds a ton of variance, and that makes the benchmark unreliable.

Should definitely do a test like this! But it'd be a performance soak test or something. Different class of thing. Happy to talk more about it if there's interest.

[alexanderbez]

The benchmarks on these varying sample sizes will give us insights into the major bottlenecks. As we discover those, we'll continue to refine existing and introduce new benchmarks into those bottleneck paths to get better understanding. So having these benchmarks on large samples will give us valuable insights.

[peterbourgon]

If you mean a benchmark in a broader sense, and not a go bench benchmark, then all good.

[alexanderbez]

I do mean go bench as a jumping off point.

[peterbourgon]

So go bench is built on the assumption that the code you run b.N times in a BenchmarkXxx function will be be quick (typically sub-millisecond) and consume basically the same amount of CPU and memory with each iteration. If the code under observation, like the specific CPU instructions that are called, can be substantially different from run to run, or if each run takes more than a second or two to finish (at worst!), or especially if the code does any kind of IO (disk, network, etc.) then go bench won't produce valid results. So if you can load those GBs of data up and benchmark sufficiently isolated parts of the state infrastructure then SGTM, but an e.g. iterator across all of Terra's pools (or whatever) isn't something you can run through go bench 😉

[alexanderbez]

We will find out 😉

Thanks for your tremendous valuable input as always 🙏

[facundomedica]

I had posted this in the wrong place
Final results:

go test -benchmem -timeout=120h -benchtime=120s -run=^$ -bench "^(BenchmarkQueryBalance|BenchmarkQueryAllBalances|BenchmarkQueryVotes|BenchmarkQuerySigningInfos|BenchmarkQueryValidatorDelegations)$" -memprofile memprofile.out -cpuprofile profile.out cosmossdk.io/simapp

goos: linux
goarch: amd64
pkg: cosmossdk.io/simapp
cpu: AMD Ryzen 9 5950X 16-Core Processor            
BenchmarkQueryBalance-28                 	66626565	      1993 ns/op	    1248 B/op	      23 allocs/op
BenchmarkQueryAllBalances-28             	 5418576	     26516 ns/op	   13880 B/op	     194 allocs/op
BenchmarkQueryVotes-28                   	   46556	   3026516 ns/op	 2882343 B/op	   58778 allocs/op
BenchmarkQuerySigningInfos-28            	  162439	    835128 ns/op	  830759 B/op	   14910 allocs/op
BenchmarkQueryValidatorDelegations-28    	      60	2359649179 ns/op	1735087374 B/op	28170831 allocs/op
PASS
ok  	cosmossdk.io/simapp	31174.6

The zip file contains both memory and cpu profiles
profiles.zip

[facundomedica]

And here we have the results of the same queries but through the HTTP interface: https://drive.google.com/file/d/1O1AB-rM095zj9to3F05M2qlhqbJ3I7PJ/view?usp=share_link

The .bin file is a result from vegeta.