Run Daily Benchmark Script Automagically

[jdjaustin]

Issue summary

Tasks for running benchmark script automatically:

• configure a droplet (detailed requirements listed below) that is automatically created daily and destroyed (destruction can either be triggered by the script or workflow, we may need to evaluate which would be best) after successful run of the script
• move the benchmark script to forest-iac and deploy
• configure droplet to export the CSV results to a location off-droplet
• configure alerts if the script fails for any reason (including during setup and installation of pre-reqs)
• test that everything works properly (creation/deletion of droplet, script runs or alerts on failure, CSV exports correctly)

Other information and links

We will need to automatically: create the droplet, install the pre-reqs as outlined in the README.md, run the benchmarks script (currently in the Forest repo here, needs to be moved to forest-iac as part of PR for this issue), export the CSV that is generated (possibly where our snapshots go?), and destroy the droplet upon successful run (or alert on failure without destroying the droplet to allow troubleshooting of the issue).

We need a machine with a dedicated CPU so the results are reproducible. A 320GB disk was sufficient to run the mainnet tests, but we may need more storage in the near future as features are added to the script. We have tested on both Ubuntu and Fedora droplets; as most of the existing IAC infra is Ubuntu, we can go with Ubuntu if we want to be consistent.

[lemmih]

Blocked by this issue: ChainSafe/forest#2208