Refactor e2e test

[2opremio]

• To reuse artifacts, run e2e sequentially, in the same CircleCI task.
• Support running e2e test locally (both in Linux and Darwin systems) through make e2e. An existing Kubernetes cluster will be used if available, otherwise a cluster will be created with Kind.

Supersedes #1895

[2opremio]

The e2e-testing testing step doesn't exist anymore, so we should considered this build as green 💚

[hiddeco]

I tried to run this on my machine and I noticed a couple of things.

1. If you have an existing minikube instance running which already has the CRD installed, it will fail.
2. After I created a fresh minikube instance and ran make e2e again, it seems to fail on the Installing Flux with Helm step.

   <snip>
   deployment "gitsrv" successfully rolled out
   >>> Installing Flux with Helm
   Error: release flux failed: timed out waiting for the condition
   
   Running deferred items, please do not interrupt until they are done:
   deferred: rm -f /home/hidde/go/src/github.com/weaveworks/flux/test/e2e/id_rsa /home/hidde/go/src/github.com/weaveworks/flux/test/e2e/id_rsa.pub
   deferred: kubectl delete namespace flux-e2e
   namespace "flux-e2e" deleted
   deferred: helm reset --force
   Tiller (the Helm server-side component) has been uninstalled from your Kubernetes Cluster.
   deferred: kubectl delete clusterrolebinding tiller-cluster-rule
   clusterrolebinding.rbac.authorization.k8s.io "tiller-cluster-rule" deleted
   deferred: kubectl --namespace kube-system delete sa tiller
   serviceaccount "tiller" deleted
   make: *** [Makefile:59: e2e] Error 1
   

3. The CRDs aren't truncated when it fails, making it hard to rerun the e2e tests given point 1.

[2opremio]

1. If you have an existing minikube instance running which already has the CRD installed, it will fail

helm install fails if they exist, and I thought preventively removing them is too invasive.

2. After I created a fresh minikube instance and ran make e2e again, it seems to fail on the Installing Flux with Helm step.

Huh, that's odd, I tested it both with Kind and my local Docker for Mac cluster. I will install minikube and give it a try.

3. The CRDs aren't truncated when it fails, making it hard to rerun the e2e tests given point

This means helm fails but gets to create the CRDs, right? I will look into this.

[hiddeco]

Helm install fails if they exist, and I thought preventively removing them is too invasive.

Removing them would probably be too invasive, maybe we can detect them and add a --no-crd-hook to helm install?

This means helm fails but gets to create the CRDs, right?

Correct, please note that purging a release with helm del --purge will not remove any CRDs installed by the chart.

[2opremio]

Removing them would probably be too invasive, maybe we can detect them and add a --no-crd-hook to helm install?

I didn't know about that, will do.

[2opremio]

Huh, that's odd, I tested it both with Kind and my local Docker for Mac cluster. I will install minikube and give it a try.

@hiddeco I tried with a Minikube instance in my Mac and it worked just fine, could you please try again?

[squaremo]

Hrm. Part of my aim in #1895 was to make it easier to add more tests, by keeping the set-up separate from the test (and to some extent modular). What was the motivation for lumping them all together?

[2opremio]

Hrm. Part of my aim in #1895 was to make it easier to add more tests, by keeping the set-up separate from the test (and to some extent modular). What was the motivation for lumping them all together?

It simply made the cleanup much easier. I can split them up, but I would like to drive that by an additional test. Would you mind if we do that later?

[squaremo]

I can split them up, but I would like to drive that by an additional test. Would you mind if we do that later?

How much do you reckon would that end up looking like the prior state (or that in #1895)? If "not much", then OK sure let's do it later. If "somewhat", then I feel like that particular change in this PR is taking a step backwards, for a short term convenience.

I suppose that boils down to: is it better to roll forward from changes made to the bits of script here, or port those changes back to the scripts as they were. I have no especial attachment to the work in #1895 except in so far as it made it possible to run some a subset of the setup scripts.

[2opremio]

@squaremo no strong opinions, the hardest part by far here was reusing the artifacts from the build in the e2e test (which isn't directly related to the scripts, it was mostly moving everything to a machine executor since Kind needs that). Also cleaning things up during local execution was tricky.

I think splitting things up again shouldn't be too hard. It may end up looking (in structure) like before, but I am not sure that should be a show stopper. I think the crucial bit is what parts we want to reuse between tests, which I want to drive by adding actual tests.

[squaremo]

I think splitting things up again shouldn't be too hard. It may end up looking (in structure) like before, but I am not sure that should be a show stopper. I think the crucial bit is what parts we want to reuse between tests, which I want to drive by adding actual tests.

👍 You're in the best position to make that call, so all good by me

[2opremio]

@hiddeco Can I have a rubber-stamp? (I will remove the e2e-testing check after merging)

[hiddeco]

(I will remove the e2e-testing check after merging)

I was waiting for this before stamping it, there you go 📮