Post-open-source CI proposal

[nomeata]

I spend some more thought about our dev infrastructure once we have open sourced Motoko.

Whither nix?

Given the amount of work that went into our nix build system, I take it as granted that we need to continue using it. So we need a CI system that supports that for Motoko (even if we may take other decisions for other repositories).

Because we want Motoko to be reasonably self-determined, and nix is phased out elsewhere, we’ll have to maintain the nix setup ourselves (but we have done so in the past as well). Who knows, maybe the community will help?

Nix cache

Developing with our nix setup really requires access to a warm nix cache that has all our build tools. Because we want Motoko development to be possible for people without access to our internal infrastructure, and to separate this from possible private data, we cannot use the company-internal nix cache.

Here natural choice is https://cachix.org/, a nix cache hosing solution. I have used it for my own open source projects and am satisfied.

We could use the “Free for Open Source” option with 5GB of storage and it might be enough, although I would suggest we do pay for at least the starter tier with 20GB for $40/month.

Action items:

• Get approval on this expense
• Someone who can pay needs to sign up
• We set up CI (see below) to fill that cache.

CI system

Additionally, we need a CI system (i.e. something that monitors the repo, runs builds, reports status and uploads artifacts to the cache). Again, we cannot continue to use our internal hydra installation (because it is internal and we are phasing out hydra use). I see the following options:

• Plain Github Actions (or a similar generic CI service like Circle CI)

  This is easy to set up, the machines are generously sponsored by Github Microsoft, and provide both linux and darwin builds. It is possible to upload artifacts to the Cachix cache.

  The downside is that this is isn’t really “nix-aware”: If you have to concurrent jobs that both are about to build the same job, both will. Smart parallelism is hard to get. The status is displayed rather bluntly.

• Hercules CI

  This is a commercial service, again free for open source, that provides dedicated nix building. Think of it as hydra, but more modern, actively supported, more features and a modern UI.

  Yesterday, I set it up for one of my open source projects with a similarly complex nix setup as Motoko, and it mostly worked fine.

  The main issue with Hercules CI is that you have to run your own runner (called “agent”). This is rather simple (install a program, configure three files), but it is a bit of work. And we’d need at least two (linux and darwin). They provide ready to run terraform and nixops configurations, so assuming these installations are as low-maintenance as I hope, I expect that our IDX team could handle the additional work (especially if they can stop worrying about hydra then).

  Alternatively, it might suffice if they provision two suitable machines and give us login access, and we take care of running the agent.

Action items:

• Find out if IDX is able to provide two builder machines
• Find out if IDX is happy to run the hercules-ci agents on them, or if we should do that.

Summary

We can probably keep our setup even as we make Motoko open source and divorce it further from our internal infrastructure. We are not blocked:

• We can start with the open source cachix offering (I hope 5GB suffice), and upgrade to the paid plan once approved.
• We can start with using Github Actions (which we have set up already a year ago in Nix: no drun by default, and build with Github Actions #1224), and switch to the more suitable Hercules CI once we have figured out the runners. (Maybe I’ll use my server initially and someone with a mac in a corner can offer that – making it feel even more like an typical open source project ;-) …)

[nomeata]

Hmm, I forgot something: Our CI jobs do fancy stuff like

• For PRs, take the base commit into account and do differential CI
• Post comments depending on build results
• act differently depending on which labels are set
• push releases for tagged revisions

This would all be possible with GitHub Actions, and maybe harder with the less configurable Circle CI. I am unsure now. I wonder if there is a good way to mix and match?

Or just start with GitHub Actions.

[crusso]

Thanks Joachim, this looks great!

[nomeata]

I am using ic-hs as a guinea pig for experimenting with cachix + Github actions in dfinity/ic-hs#2. Looks good so far, despite a custom rust compiler etc. Maybe that is the better first option.

[nomeata]

Work on the CI setup in #1224. Only simple building an nix cache filling for now, but it’s a start.

[nomeata]

Note to myself: https://github.com/marketplace/actions/create-or-update-comment might be a start to recreate our performance diff reporting feature. https://github.com/marketplace/actions/create-or-update-comment#where-to-find-the-id-of-a-comment seems to be precisely what we need. Or start building it with github script (https://github.com/actions/github-script#comment-on-an-issue)

[nomeata]

I’ll close this, given that #1224 is merged. More steps tracked in #2565.