Support full compatibility test runs through a GitHub action

[chatton]

Description

This PR adds a workflow which will let us specify a release branch, and then run a full set of predetermined tasks using the tip of the release branch. e.g. release/v5.0.x. This PR adds the bare minimum number of tests (1!)

In a follow up PR, we can add the full set of tests we want to run for compatibility testing (and the additional release branches)

The json configuration files feed directly into the github action which specifies arguments to be provided to make e2e-test

This approach has the downside of manually maintaining sets of tests and versions, but I don't think this should be too much of a problem as this won't change that frequently. We could update the test matrix coming up to a release. I'm happy to hear other suggestions though!

See this workflow as an example.

If we take this approach, we may be able to remove this approach entirely.

closes: #2368

---

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

• Targeted PR against correct branch (see CONTRIBUTING.md)
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Code follows the module structure standards.
• Wrote unit and integration tests
• Updated relevant documentation (docs/) or specification (x/<module>/spec/)
• Added relevant godoc comments.
• Added a relevant changelog entry to the Unreleased section in CHANGELOG.md
• Re-reviewed Files changed in the Github PR explorer
• Review Codecov Report in the comment section below once CI passes

[chatton]

this will be populated with every permutation of test we want to run with release-v5.0.x in a follow up pr (along with corresponding files for other release branches)

This is just a sample to show how the full test suite will be run.

[chatton]

this workflow won't become available until it is merged to main, the sample workflow I linked hard coded values rather than receiving them from UI

[crodriguezvega]

So here we would add other release branches to have them as options in the GitHub UI, right?

[chatton]

yes, anything in this list will show up in a drop down menu. There's nothing stopping us to add a new field which allows anything, but I think the dropdown is a nice UX. Perhaps we could add an optional override which would take precedence in the case of wanting to run a custom image.

[crodriguezvega]

🪄

[crodriguezvega]

If we want to see the results of the tests we would need to go to the e2e-tests job and check at the bottom of Run e2e test, right?

Maybe we can think of creating a nice visual dashboard for this?

[chatton]

@crodriguezvega the view we would have would be essentially the same as what it looks like on a regular PR run e.g. this

One nice-to-have would be uploading the logs of failed tests which we currently don't do.

[colin-axner]

Amazing! Can't wait to give it a shot!