Load data using snapshot repository during benchmark runs

[rishabh6788]

As of now for each and every benchmark run the data is downloaded locally and then indexed into the cluster before the search queries are run, this process takes the most time in the overall benchmark process.
Explore ways to be able to restore the data into the indices from a snapshot repository to speed up the process.
This will also provide more consistent results with search queries.

Pre-requisites:

• Create a S3 snapshot in testing infra aws account and add snapshots of all the available opensearch-benchmark-workloads.
• Add support in opensearch-cluster-cdk to install repository-s3 plugin and add S3 snapshot bucket IAM permission.
• Add support in python workflow to accept --include-tasks, --exclude-tasks and --test-procedure.

Acceptance Criteria:

• The workload data should not get downloaded and indexed while using restore from snapshot test procedure.
• The # of segments should remain constant for each workload indices.
• The search query latency variance should not exceed 5%.
• The benchmark runtime should reduce by 50%.

[Divyaasm]

@rishabh6788 , could you add more details like acceptance criteria to the issue.

[rishabh6788]

Following are the task break-downs and their status:

• Create s3 bucket for snapshot-repository: This is completed, the snapshot bucket will reside in the same account where the test clusters are brought up for benchmarking, this will ease the process of registering and fetching the snapshot without requiring to import aws keys and tokens. The attached role to opensearch ec2 instances will have required access to be able to fetch the snapshot.
• Create snapshots of opensearch-benchmark workloads (In-progress): Currently working on indexing the opensearch-benchmark workloads data into a test cluster and then creating snapshots of them with different configurations, such as default https_logs snapshot with default # of clients and another snapshot with just 1 client to be able ingest data linearly. This will be an ongoing process as and when demand arises for the a new workload snapshot. We will start with nyc_taxis, http_logs and pmc.
• Create new workload test procedures (Not started): Once the snapshots have been created, will work on creating custom test-procedures that will use the snapshot to index the data and then run search queries. The custom test-procedure should be customizable, i.e., the user should be able to choose which snapshot to use and what should be the index configuration while restoring the data.
• Add snapshot support in opensearch-cluster-cdk (Not started): We need add respository-s3 plugin installation as default action and also make changes to ec2 instance IAM profile to be able to fetch snapshot from s3 bucket.
• Add support in main nightly benchmark python workflow (Not started): Finally add the necessary command line parameters support in the main workflow to be able to utilize above mentioned changes. This will also require changes in our nightly run jenkinsfile and corresponding groovy library

@bbarani @getsaurabh02 @rishabhmaurya @Pallavi-AWS

[rishabh6788]

All the required milestones have been completed and nightly benchmarks are now running using snapshots.