[FEATURE] Onboard distribution level BWC bundle test

[zelinh]

Currently security repo has BWC test on plugin level and only tested within the repo. We have a framework for BWC distribution level test available and you could utilize it and enable distribution level BWC test.

With this framework, you would be able to spin up the test cluster with the latest distribution bundle that engineering effectiveness team generated that installs all the other components. This test will be more comprehensive as it will test the exact product we ship to the customers.

Onboarding this distribution level BWC test can also enable engineering effectiveness team to run BWC tests every time ee team generates a new bundle distribution across each of the plugin through CI

More onboarding info can be found here:
https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#bwc-tests-on-distribution-bundle-level

What solution would you like?
Similar to what we did to the anomaly-detection plugin. opensearch-project/anomaly-detection#766
Enable bundle level BWC test based on the BWC test you already have.

[stephen-crawford]

[Triage] Hi @zelinh, thank you for filing this issue. Based on the linked campaign it seems like this is coming from a build team effort. Can we expect a build team member to be opening a pull request to make this change or are you looking for each repo to swap to the new framework themselves? We can close this issue when we have swapped over to the new framework so going to mark as triaged.

[zelinh]

Hi @scrawfor99, we are looking for each plugin team to onboard to this distribution level BWC tests since we have an existing framework for this. It's not swapping; currently the BWC tests most teams running are plugin level tests which was testing the plugin within cluster with only core and this particular plugin.

We want each team to onboard to this distribution level BWC tests framework as well so that we are testing in this case, security plugin with a whole bundle cluster.

[stephen-crawford]

This is not going to be complete by 2.13 considering the lack of PR. I am removing the tag.

[zelinh]

Can we get some traction on this issue? This BWC tests request would be a future release blocker.

[zelinh]

@stephen-crawford Do we have a plan to onboard to this distribution level BWC test? We are planning to add BWC tests as mandatory exit criteria in future releases.