Make CI and local test workflows the same

[micahhausler]

In the last few Tink PRs I've made, one major point of difficulty has been that make targets don't fully reflect what gets run in CI. I can have a branch where running make test && make verify works locally, but CI can still fail for various reasons.

I've filed one-off issues or questions on PRs about this past (ex: #556, comment on #563), but it seems to me that a stated policy for Tinkerbell (not just this repo) of something like "Tests run in CI should be easy and straightforward to run locally." This may be as simple as enforcing that only make commands are invoked in CI, aside from some CI build cache setup.

Expected Behavior

As a developer making changes to Tinkerbell, I expect that I can run validations locally that my code is correct, and expect that CI will pass.

Current Behavior & Possible Solutions

The current test setup behavior relies on Nix shell and also runs some one-off commands.

• Nix seems great, but not everyone has that setup locally. I think a make target for setting up a nixos container + a tooling/build cache volume could lower the barrier to new developers
• Several CI steps are either partially duplicated from make targets, or call one-off make targets
  • go clean -testcache
  • go test
  • make bin/gofumpt
  • ./ci-checks.sh
  • make generate-manifest includes the nodeJS-based prettier
    

Context

In the 12/9/21 Tinkerbell community meeting, this came up as a discussion point.

Your Environment

Developers are probably using either macOS, Linux, or Windows, and it'd be nice to accommodate all three, or state which ones we support.

[displague]

Something that I have found useful for testing GH Action CI locally is https://github.com/nektos/act#example-commands

(This is not a direct response to the issue.)

[displague]

I agree that the current state is not idea. I don't use nix and I'd rather not have to think about it. I like make and obvious targets. I build and test from Mac and Linux and don't want to hit hurdles when I switch environments.

I like when GH actions do GH specific things (annotating lines of code, adding coverage reports to the UI, doing robot work).
I don't like when GH actions or local tests take extra time to setup build environments (especially if this is done frequently, not cached or otherwise persisted).

[chrisdoherty4]

@micahhausler If we head down a container route is there really a need for nix? Containers give you everything nix does, just in a different and arguably more well known way.

[chrisdoherty4]

We've made significant headway in improving this with recent changes such as #656.

There's more to do but most of the initial points raised are fixed. Will raise separate more specific issues as they come up.