Support running Integ and BWC tests for multiple test manifests

[gaiksaya]

Is your feature request related to a problem? Please describe

With more and more features coming in, along with default test manifest that contains everything set to default for opensearch core, we saw more test manifest being added with non-default parameters. Example: test manifest with concurrent search enabled, segment replication enabled, etc.

Currently we do not have a mechanism to pass multiple test manifests at the jenkins job level, when integration and BWC tests are triggered via distribution pipeline. Instead we are adding these manifests to run via a cronjob.

Along with this, the GH issue creation do not specify what manifest was used to run the tests.

Describe the solution you'd like

Ability to pass and run integration and BWC tests for multiple test manifests.

Describe alternatives you've considered

Continue using cronjobs but this is inefficient as check-for-build jobs triggers the distribution build and tests only when there are changes in components across mentioned GitHub reference avoiding unnecessary rebuilds.

Additional context

• The existing distribution build TEST_MANIFEST parameter should support taking multiple test manifest as part of the job execution, having this will allow triggering integ/BWC test job with the same distribution build number rather than using latest artifact.

Acceptance criteria:

• Tweak the existing TEST_MANIFEST parameter in distribution build jenkinsfile to accept multiple manifest files: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L39
• Tweaking the function of integ-test trigger, so that a for loop block can trigger multiple integ-test runs based on the provided manifest: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L958
• All the integ-test runs should use the same artifacts for cluster building and testing.