Support E2E Software Upgrade Tests

[chatton]

We should be able to ensure that we can upgrade from the latest release of a release branch to the code of a an arbitrary branch/PR/release candidate and ensure that nothing breaks in doing so.

We can achieve this with the following steps.

• Add a manually triggerable workflow (workflow_dispatch) to the main branch that does the following
• Determine the latest release in for that branch.
  We could determine this by doing something like this from a workflow.

   VERSION="3.1"
   git tag -l | grep "v${VERSION}" | sort -V | tail -n 1
   # v3.1.1

• Specify a new environment variable for the tests, UPGRADE_FROM or similar to provide this version to the tests.
• Perform an automated upgrade from that version, to an image built from the current code. And run our test suite.
  This should effectively perform follow the steps outlined here but use the ibctest framework to broadcast messages instead of using cli commands directly.
• Perform any post upgrade checks to verify that the upgrade was successful. The previous test can be generalized and run only if the before and after versions differ. This way we can turn any existing test into an upgrade test.

Note: The test code will stay in the main branch, the workflow will pull tags that have been built and pushed from tasks that ran during the release process of the other release branches. This way all the test code lives in one place. We can specify which tests should run on which given release if required.

[colin-axner]

We should setup a conventional way to test state persistence/migrations as part of this issue, but first I'd like to just see a test go from one major version to the next

[colin-axner]

https://github.com/cosmos/cosmos-sdk/blob/main/x/upgrade/types/proposal.go

[colin-axner]

I guess the new action item is ensuring there's a test for v5 -> v6?

[chatton]

@colin-axner right this issue's scope is maybe ever expanding. Maybe we can close this issue as we have an upgrade test that we can run. We can create issues for individual upgrade cases we want to handle. WDYT?

[colin-axner]

#2188 can be used for v5 -> v6 upgrades