[Meta] Test Orchestrator

[ashwin-pc]

RFC to the Orchestrator remote test workflow feature - #5392

Design POC - #5526

README doc - opensearch-project/opensearch-dashboards-functional-test#1119

Priorities

P0

• Add test orchestrator script to trigger remote workflow
• Add Github workflow for Test Orchestrator in OpenSearch Dashboards
• Execute script from the release testing workflow of the functional test repo
  • Ensure that this execution is non-blocking to the execution of integtest.sh. In a fire and forget manner, do not wait for the results.
  • For an initial deliverable we will have results showing that these tests are being executed. If it does not fire and forget then it will take in the allotted execution time (roughly 2 hours) and whatever tests we execute will add to the total test runtime.
• Clean up and add cypress configurations to test folder of OpenSearch Dashboards
• Onboard lightweight plugin so that they can call their tests within their repo and still be executed if infra onboard: Onboard plugin to orchestrator workflow opensearch-dashboards-functional-test#1137
  • Select a plugin that meets the requirements
• Update cypress test workflow for OpenSearch Dashboards to execute the local cypress tests (Issue to address release_cypress_workflow PR comments #5892)
  • Will run when pull requests are made for the new code
  • Then the remote test workflow will all for the execution of tests that includes the release builds that can be downloaded and executed and provided verification
• Standardized CI groups before adding these tests. If the framework does not build with CI groups in mind then tests will be added and ran sequentially. If we add thousands of tests this will take a lot of time.(Issue to address release_cypress_workflow PR comments #5892)
• Absorb @opensearch-dashboards-test/opensearch-dashboards-test-library into OpenSearch Dashboards(Issue to address release_cypress_workflow PR comments #5892)
  • Potentially @osd/test
  • Everybody has to pull down OpenSearch Dashboards for their component. We can add more content and stabilize the helper functions instead of relying on an external package to help make these tests consistent and stable
• Blog
  • Talks about this work
  • Remaining work
  • Gives the ideal state and best outcome

P1

• Extend testing framework to make it easier for plugins to verify cross compatibility testing with the plugin
  • Plugins can easily reference this package as it is a part of OpenSearch Dashboards
  • Will install their plugin into the version of OpenSearch Dashboards and run a basic suite of tests for the version
  • Additional tests can be passed from the plugin
  • Outputs successful or failed tests
  • Plugins can decide if they want to indicate they are fully supported or degraded
  • Create 3 example plugins
    • Positive (version available for engine and works)
    • Negative (version not available for engine for cross compatibility)
    • Partially positive (version kind of works)
  • Hook into the test orchestrator
• Work with infra to unblock onboarding the ability to listen to the remote test runners
  • Get credentials
• Campaign to add plugins to utilize workflow
• Migrate more tests from Selenium to Cypress