Helm upgrade and SSL certificates

[cyriltovena]

When updating using helm upgrade the certificates for the admission controller gets regenerated but the controller pod doesn’t get restarted and thus keep the old certificates which causes this error

GameServer instances failed (): Internal error occurred: failed calling admission webhook 
"mutations.stable.agones.dev": Post https://agones-controller-service.agones-
system.svc:443/mutate?timeout=30s: x509:certificate signed by unknown authority 
(possibly because of "crypto/rsa: verification error" while trying to verify candidate 
authority certificate "admission-controller-ca")

using helm upgrade --recreate-pods fixes the issue but we needs to document it.

@markmandel are we ok in recreating the controller pod at each upgrade ? Or do you have a better idea ?

[markmandel]

Copy pasting chat transcripts, for posterity:

Mark Mandel [8:13 AM]
Also need to respond to your comment re: the internal ssl cert

I've run into that before as well - I usually hand delete the pod - but it's something we should discuss overall

Cyril Tovena [8:46 AM]
for now in my MR I forced the recreation of the pods
and it fixes the issue
I don't think the fix is easy

Mark Mandel [8:47 AM]
it's really only a development issue
in prod, you'll get a new tagged image, so the original pod will get deleted and replaced

Cyril Tovena [8:48 AM]
true unless you just want to change something else

Mark Mandel [8:48 AM]
oh true!

Cyril Tovena [8:48 AM]
but it's minor you're right I would not spend a lot of time for now

Mark Mandel [8:48 AM]
Is there a way in helm to only generate the cert if it's not there already?

Cyril Tovena [8:48 AM]
that's why I went for a recreate
I did not search more info, they have a solution with sha

Mark Mandel [8:49 AM]
yeah, we could also add a faq/troubleshooting guide to the development guide for this

Cyril Tovena [8:49 AM]
exactly
I think we can live with this, let's just mention it somewhere

Mark Mandel [8:49 AM]
Actually - having a troubleshooting section would be good anyway

Cyril Tovena [8:49 AM]
definitively

[markmandel]

@Kuqd did your recent #315 solve this problem as well, since you added --recreate-pods to the install target?

[cyriltovena]

Yes indeed, but it’s fixed only if you use the make target to install!

[cyriltovena]

I'd like to explore this https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change

[cyriltovena]

so I could use an annotation to force the restart of the pod, but the drawback is I need to merge the controller.yaml and admissionregistration.yaml to access the generated cert.(I tried to use include function but it gets re-evaluated.)

At this point I realized that a timestamp as annotation would have the same result, since certificates are regenerated all the time. (There is no way to generate only at install time).

So I think we should add a timestamp in the controller annotations(only if cert are generated), as this would be less error prone than a troubleshooting section and recommend for production user in the helm documentation to generate their own certificates, in that case the problem is non-existent and restart are avoided.

WDYT ?

[markmandel]

This idea sounds good to me!