Generate Daily Benchmark Results (#2367)

.gitignore

@@ -14,3 +14,7 @@ genesis.json
 node/rpc-api/static/ast.ron
 documentation/book/**/*
 node_modules
+/scripts/benchmark_db/.bundle/
+/scripts/benchmark_db/vendor/
+/scripts/benchmark_db/*.md
+/scripts/benchmark_db/*.csv

scripts/benchmark_db/.rubocop.yml

@@ -0,0 +1,6 @@
+AllCops:
+  NewCops: enable
+Layout/LineLength:
+  Max: 120
+Metrics/MethodLength:
+  Max: 20

scripts/benchmark_db/Gemfile

@@ -4,5 +4,14 @@ source 'https://rubygems.org'
 
 git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 
+gem 'concurrent-ruby', '~> 1.2', '>= 1.2.2'
+gem 'csv', '~> 3.2', '>= 3.2.6'
 gem 'deep_merge', '~> 1.2', '>= 1.2.2'
+gem 'fileutils', '~> 1.7'
+gem 'logger', '~> 1.5', '>= 1.5.3'
+gem 'open3', '~> 0.1.2'
+gem 'optparse', '~> 0.3.1'
+gem 'set', '~> 1.0', '>= 1.0.3'
+gem 'tmpdir', '~> 0.1.3'
 gem 'toml-rb', '~> 2.2'
+gem 'tomlrb', '~> 2.0', '>= 2.0.3'

scripts/benchmark_db/Gemfile.lock

@@ -0,0 +1,36 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    citrus (3.0.2)
+    concurrent-ruby (1.2.2)
+    csv (3.2.6)
+    deep_merge (1.2.2)
+    fileutils (1.7.0)
+    logger (1.5.3)
+    open3 (0.1.2)
+    optparse (0.3.1)
+    set (1.0.3)
+    tmpdir (0.1.3)
+      fileutils
+    toml-rb (2.2.0)
+      citrus (~> 3.0, > 3.0)
+    tomlrb (2.0.3)
+
+PLATFORMS
+  arm64-darwin-21
+
+DEPENDENCIES
+  concurrent-ruby (~> 1.2, >= 1.2.2)
+  csv (~> 3.2, >= 3.2.6)
+  deep_merge (~> 1.2, >= 1.2.2)
+  fileutils (~> 1.7)
+  logger (~> 1.5, >= 1.5.3)
+  open3 (~> 0.1.2)
+  optparse (~> 0.3.1)
+  set (~> 1.0, >= 1.0.3)
+  tmpdir (~> 0.1.3)
+  toml-rb (~> 2.2)
+  tomlrb (~> 2.0, >= 2.0.3)
+
+BUNDLED WITH
+   2.4.6

scripts/benchmark_db/README.md

@@ -1,49 +1,116 @@
 # Forest benchmark db script
 
-This script is here to help with testing of Forest db backends and their
-configuration.
+This script was developed to help with testing of Forest db backends and their
+configuration; the script now also allows benchmarking ("daily" benchmarks) of
+Forest and Lotus snapshot import times (in sec) and validation times (in
+tipsets/sec).
 
 ## Install dependencies
 
-You will need to install Ruby first. Then go into `benchmark_db` and type:
+[Install Ruby](https://www.ruby-lang.org/en/documentation/installation/) first.
+Then go into `scripts/benchmark_db` and execute the following commands:
 
 ```
+$ bundle config set --local path 'vendor/bundle'
 $ bundle install
 ```
 
+Note: depending upon your Ruby installation, it may be necessary to execute
+`gem install bundler` first. In case of any issues with "native extensions"
+during `bundle install` on a \*nix machine, it may also be necessary to execute
+`apt-get update && apt-get install -y build-essential ruby-dev`.
+
+The daily benchmarks also require the installation of
+[aria2](https://github.com/aria2/aria2) and
+[zstd](https://github.com/facebook/zstd), as well as dependencies required for
+the installation of [Forest](https://github.com/ChainSafe/forest) and
+[Lotus](https://github.com/filecoin-project/lotus) (note that the script handles
+installation of the Forest and Lotus binaries).
+
 ## Run benchmarks
 
-You need to run the script at the root of the repository. Ie:
+Run the script at the root of the repository. I.e.,:
 
 ```
-$ ./benchmark_db/bench.rb 2369040_2022_11_25t12_00_00z.car
+$ ./scripts/benchmark_db/bench.rb <path to snapshot> <optional flags>
 ```
 
-You can create a selection of benchmarks using the `--pattern` flag. If used in
-conjunction with `--dry-run` you will see what commands will be run:
+If the user does not specify a path to a snapshot, the script will automatically
+download a fresh snapshot, then pause for 5 minutes to allow the network to
+advance to ensure that enough time will be spent in the `message sync` stage for
+proper calculation of the validation time metric. Also note that if `--chain` is
+specified, the user must provide a script matching the specified `<chain>` (the
+script defaults to `mainnet`, so if `--chain` is not specified, provide a
+`mainnet` snapshot).
+
+If the `--daily` flag is included in the command line arguments, the script will
+run the daily benchmarks specified earlier; otherwise the script will run the
+backend metrics.
+
+On many machines, running the script with `--chain mainnet` may require more
+space than allocated to the `tmp` partition. To address this, specify the
+`--tempdir` flag with a user-defined directory (which will automatically be
+created if it does not already exist).
+
+To create a selection of benchmarks, use the `--pattern` flag (current defined
+patterns are `'*'`, `'baseline'`, `'jemalloc'`, and `'mimalloc'`). Using
+`--dry-run` outputs to the terminal the commands the script will run (without
+actually running the commands):
 
 ```
-$ ./scripts/benchmark_db/bench.rb 2369040_2022_11_25t12_00_00z.car --pattern 'baseline*,paritydb' --dry-run
-Running bench: baseline
+$ ./scripts/benchmark_db/bench.rb <path to snapshot> --chain calibnet --pattern jemalloc --dry-run
+(I) Using snapshot: <path to snapshot>
+(I) WORKING_DIR: <generated directory>
+
+(I) Running bench: jemalloc
+(I) Building artefacts...
+(I) Cloning repository
+$ git clone https://github.com/ChainSafe/forest.git forest
+(I) Clean and build client
 $ cargo clean
-$ cargo build --release
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
-Wiping db
+$ cargo build --release --no-default-features --features forest_fil_cns,jemalloc
+$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
+$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load=true --height <height>
+(I) Clean db
+$ ./forest/target/release/forest-cli -c <path>
+toml db clean --force
 
-Running bench: baseline-with-stats
+(I) Wrote result_<time>.md
+```
+
+```
+$ ./scripts/benchmark_db/bench.rb <path to snapshot> --chain calibnet --dry-run --daily
+(I) Using snapshot: <path to snapshot>
+(I) WORKING_DIR: <generated directory>
+
+(I) Running bench: forest
+(I) Building artefacts...
+(I) Cloning repository
+$ git clone https://github.com/ChainSafe/forest.git forest
+(I) Clean and build client
 $ cargo clean
 $ cargo build --release
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
-Wiping db
+$ ./forest/target/release/forest-cli fetch-params --keys
+$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
+$ ./forest/target/release/forest --config <tbd> --encrypt-keystore false
+(I) Clean db
+$ ./forest/target/release/forest-cli -c <path> db clean --force
 
-Running bench: paritydb
-$ cargo clean
-$ cargo build --release --no-default-features --features forest_fil_cns,paritydb
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import
-$ ./target/release/forest --config <tbd> --encrypt-keystore false --import-snapshot <tbd> --halt-after-import --skip-load --height 2368640
-Wiping db
+(I) Running bench: lotus
+(I) Building artefacts...
+(I) Cloning repository
+$ git clone https://github.com/filecoin-project/lotus.git lotus
+(I) Clean and build client
+$ make clean
+$ make calibnet
+$ ./lotus/lotus daemon --import-snapshot <tbd> --halt-after-import
+$ ./lotus/lotus daemon
+(I) Clean db
+
+Wrote result_<time>.csv
 ```
 
-Benchmark results will be written to a markdown file at the end.
+As seen in these examples, if `--daily` is passed in the command line, daily
+benchmark results are written to a CSV in the current directory with naming
+format `result_<time>.csv`. Otherwise, backend benchmark results will be written
+to a markdown file with a similar naming convention.