Fix rolling update deadlock.

[d-kuro]

aws-secret-operator has deadlock occurs when rolling updating.

Details

In the leader.Become function, the operator waits until it becomes the leader.

After that, Operator is create the file /tmp/operator-sdk-ready with ready.NewFileReady().

// Become the leader before proceeding
leader.Become(context.TODO(), "aws-secret-operator-lock")

r := ready.NewFileReady()

Rolling update

In the case of the rolling update, it works as follows.

1. A new Pod is created
2. A new Pod tries to be the leader with call leader.Become function.
3. But the new Pod keeps waiting, as the old Pod is now the leader
4. If a new Pod is not the leader, file /tmp/operator-sdk-ready is will be not created
5. ReadinessProbe will not succeed without file /tmp/operator-sdk-ready
6. Deadlock ☠️ rolling updates stoped

This issue is the same as reported in the operator-sdk issue.

operator-framework/operator-sdk#920

operator-sdk fixes this bug in v0.4.0 release.

https://github.com/operator-framework/operator-sdk/releases/tag/v0.4.0

Fixes deadlocks during operator deployment rollouts, which were caused by operator pods requiring a leader election lock to become ready (#932)

This bug fix removes the readiness check.
I include the same bug fix into aws-secret-operator.

logs

$ kubectl get deploy aws-secret-operator
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
aws-secret-operator   1/1     1            1           10h

$ kubectl rollout restart deployment/aws-secret-operator
deployment.extensions/aws-secret-operator restarted

$ kubectl get rs | grep aws-secret-operator
aws-secret-operator-7d95989bb                          1         1         1       11h
aws-secret-operator-c789d48f4                          1         1         0       7m9s

$ kubectl get po | grep aws-secret-operator
aws-secret-operator-7d95989bb-h77qn                          1/1     Running   0          11h
aws-secret-operator-c789d48f4-6mnww                          0/1     Running   0          7m36s

$ kubectl logs aws-secret-operator-c789d48f4-6mnww -n pr-13773
{"level":"info","ts":1578319058.748624,"logger":"cmd","caller":"manager/main.go:27","msg":"Go Version: go1.12.5"}
{"level":"info","ts":1578319058.7495637,"logger":"cmd","caller":"manager/main.go:28","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1578319058.74957,"logger":"cmd","caller":"manager/main.go:29","msg":"operator-sdk Version: v0.1.1+git"}
{"level":"info","ts":1578319058.7497647,"logger":"leader","caller":"leader/leader.go:55","msg":"Trying to become the leader."}
{"level":"info","ts":1578319058.9293246,"logger":"leader","caller":"leader/leader.go:99","msg":"Found existing lock","LockOwner":"aws-secret-operator-7d95989bb-h77qn"}
{"level":"info","ts":1578319058.9440324,"logger":"leader","caller":"leader/leader.go:130","msg":"Not the leader. Waiting."}
{"level":"info","ts":1578319060.0760155,"logger":"leader","caller":"leader/leader.go:130","msg":"Not the leader. Waiting."}
{"level":"info","ts":1578319062.4629476,"logger":"leader","caller":"leader/leader.go:130","msg":"Not the leader. Waiting."}
...

[mumoshu]

LGTM. Thanks a lot for your contribution @d-kuro ☺️