Add devnets in CI

[clevinson]

As described in #9129, we should set up a github action that spins up a docker-based devnet (we can reuse the localnet setup in the Makefile) either just on master or every PR.

We'll need to look into lightweight hosting providers, perhaps Heroku review apps may be the way to go: https://devcenter.heroku.com/articles/github-integration-review-apps#review-apps-and-docker.

[clevinson]

Unfortunately, i'm not sure if heroku review apps actually can satisfy our needs here. Heroku in general is mostly meant for single container applications when using heroku's Container Registry and/or Docker Deploys.

Heroku does have some support for deploying multiple images as part of a larger App / Pipeline (e.g. a web & worker setup), but it seems we don't have control over what ports are exposed (see Heroku's documentation on docker deploys)... so I don't think we can feasibly have a 4-node devnet spun up via heroku at all.

One alternative that i've thought about would be using Amazon ECS (elastic container registry), together with our existing Docker Compose functionality (akin to the make localnet-start command). I've been looking at a blog post on Using docker compose with Amazon ECS, together with the documentation for Amazon ECS "Deploy Task Definition" for Github Actions.

Is using something like ECS for this overkill? Are there other alternatives we should be considering here?

I'll continue prototyping with some of this stuff on Regen Ledger (as we have the same needs there), but would be good to see what the broader SDK community things of this as well.

[aaronc]

Why is docker a requirement? We should just try the golang buildpack: https://devcenter.heroku.com/articles/go-support

[aaronc]

We want to spin up multiple nodes which should be possible in a single golang process. Maybe we can't expose enough ports idk

[clevinson]

Using docker seemed like it would be easier, seeing as we already have a perfectly suitable docker-compose.yml file ready for this use case.

Putting hte docker conversation aside, in the Heroku docs on Private Space Runtime Networking, it seems that heroku still will only expose port 80 for incoming requests, and forward those to the port designated by the $PORT env variable in the dyno process.

I've been assuming that our requirements include exposing ports for all main APIs (e.g. Tendermint RPC, gRPC, REST), as the point of having devnet in CI is to make manaul testing against a live devnet easier. I'm not sure how we can proceed then with Heroku, unless we do some workaround where we have several heroku dynos each acting like a web service listening on port 80, serving as reverse proxy to the corresponding ports (26657, 9090, 1317...) on a 4-node golang process running the testnet in a Heroku "Private Space". This seems a bit hacky. Maybe i'm missing a more obvious strategy here?

[aaronc]

Okay let's try another solution then.