CI: move to GitHub Actions?

[AkihiroSuda]

Travis often fails to report the CI status these days 😢
GitHub Actions might be more stable.

[cyphar]

I'm currently moving umoci's CI to GitHub Actions (despite my dislike for making our processes locked into GitHub). My current experience is that it's "fine", though quite a few aspects of setting up CI workflows are far more frustrating than necessary...

[cyphar]

After spending a whole day to port umoci's CI to GitHub Actions I found that it's IMHO the worst CI system I've ever used. I'm sure there are some that are worse, but every aspect of getting umoci's CI to work was incredibly painful (not to mention the end result is still incredibly slow even compared to Travis which doesn't have any of my caching optimisations). To be fair, umoci's CI is a bit more complicated than runc's but I suspect it wouldn't be much better. Even basic things like getting the logs of a currently-running action are dodgy.

After complaining about this, a colleague of mine (who's in the Kubernetes world) mentioned that CircleCI is much better designed for this kind of thing -- so maybe we should look at them instead of GitHub actions?

[AkihiroSuda]

@cyphar What are the actual problems of GH Actions?

CircleCI

It doesn't support nested VM AFAIK.
Also, its free(-as-in-beer) edition has tight limitation: https://circleci.com/pricing/#comparison-table

[cyphar]

What are the actual problems of GH Actions?

Well, here's a short list which affect umoci:

• The documentation is lacking. It took me a fair while to configure the CI, mostly because the documentation (in true GitHub fashion) doesn't provide a proper reference text for the features and semantics. You have to carefully read the getting started sections as well as the "reference" text to figure out what is going on -- and many parts of the reference text completely omit information that is pretty important (for instance, the documentation on ${{ }} doesn't describe what many of the built-in contexts mean).
  • This is compounded by the fact that many incredibly basic features like "check out the code of this repository" are not extensively documented by the Workflows documentation, so you have to look through the https://github.com/actions/checkout README in order to understand what all the parameters do (and even then the README is missing information as well).
  • Lots of the documentation links take you to the GitHub API documentation which doesn't actually answer what the link says (for instance, try to figure out what each of the possible event types mean in the on block -- all of the links take you to the GitHub API which is written from the perspective of a GitHub App and the set of events in the API are different to the ones available in Workflows).
• Performance (both start-up time as well as general performance) is worse than Travis in my experience. Right now the test umoci CI I have runs far longer than Travis (28-34 minutes versus >40 minutes) and the GitHub Actions configuration I have now uses cached build images -- when I disabled caching it took significantly more than an hour.
• The caching feature is of questionable utility since it seems like it cannot use caches uploaded during the current workflow run -- the workflow has to be finished before future jobs can make use of the cache. This means that even if a job hypothetically manages to build the image and upload it to the cache (which you cannot do manually because the cache action does uploads at the end of a job) all other jobs will repeat the same work.
  • I use the word "seems" because whether this is true is both not documented and the documentation lives in the README of the Node.js library which the Actions is implemented using. It's possible that I'm wrong, but some test runs seem to show that this is the case.
• The only way of transferring information between jobs is using artefacts (for umoci this is used for the coverage profile collation -- where all I want is to have a final fan-in job which runs an awk and shell script on some text files generated by each job), and this feature has an abundance of issues:
  • As before, the documentation is in a GitHub README.
  • You can upload to the same artefact name more than once, but doing so results in you modifying the artefact (not replacing its contents) and this feature is explicitly warned against because it can cause "corruption" (though I assume they mean there's a race condition where you can lose files added by another job). I do not understand why they added this feature if it doesn't work by design. This means you have to give a unique name to each artefact.
  • Artefacts generated during a workflow cannot be deleted during that workflow's execution (funnily enough you couldn't delete them at all until a few months ago), meaning that all of these artefacts will be displayed for download at the end of the workflow run. There are workarounds (where you have an action that creates a new workflow and triggers it after the current workflow is finished) but they would certainly cause more issues. This isn't too bad until you consider that:
    • Artefacts do not expire (or at least, they didn't according to the GitHub issues -- this is not documented anywhere -- but the GitHub API appears to have some expiry fields which I don't understand).
    • There is a per-repository limit of how much total storage can be used by artefacts, at which point uploads will fail or you will start getting charged money.

Don't get me wrong, we could just use it to run make ci. It's just that I think it's not suited to anything more complicated than that, given that all I wanted to do was to be able to stop duplicating the make unit and make ci-image steps as well as collate the code coverage over all test runs across operating systems and I ended up with a setup which seems to be slower despite doing less work. My goal was to eventually have fuzzing jobs for umoci which would dynamically update the fuzzing corpus as part of a scheduled GitHub Action -- but it seems that this would just be even more painful.

It doesn't support nested VM AFAIK.

As in vagrant doesn't work, or it's slow? (Though I guess either way that is a problem.) Does this apply to the "machine" executor?

Also, its free(-as-in-beer) edition has tight limitation: https://circleci.com/pricing/#comparison-table

Yeah I found this out as well, however since we are an open source project we qualify for more features. For umoci, the need to email them to get MacOS support is a bit odd but that doesn't affect runc.

[mikebrow]

they take PRs... people can write their own actions... just sayin

the issues I've had with actions so far are around finding good examples/docs .. esp. with the new features just added.

[mikebrow]

on the OP issue.. of reporting check status.. yeah it also happens on github actions, just not as often and when it does it's easier to restart and more integrated.

[kolyshkin]

FYI, I spent some time adding a few checks to GH actions (#2687 and #2690), and they seem OK. I wholeheartedly agree about the lack of docs, and as a whole the feature seems immature, but it looks like it's getting better.

Given the fact that travis-ci.org situation is not too clear, I guess it makes sense to diversify and use other CI platforms, including github actions.