Certificate renewal requires a restart of the gmsa pods

[pmcenery-bl]

I'm not sure if this is a feature or known issue, but when the gmsa certificate is renewed, the gmsa pods need a restart.

Background

I am using the helm chart (v0.7.2) with cert-manager v1.11.0 (but various others tried) to deploy the gmsa webhook.

How to replicate

Set a short lifetime on the certificate and see what happens when it expires. Errors appear in the logs of the gmsa pods. Any other pods trying to start report invalid certificate when calling the gmsa webhook. The solution is to restart the gmsa pod.

Workaround 1

Set a really long certificate lifetime

Workaround 2 (preferred)

Use stakater reloader to automatically restart the pod when the cert is replaced

Has anyone else had to deal with this? I just thought its worth logging before someone else runs into it and spends a bit of time trying to figure it out.

[jsturtevant]

Management and Rotation of the certs is out of scope but the webhook should be able to load new ones if updated.

The tls configuration is mounted as a secret, so any change to the secrert should be reflected in the pod:

windows-gmsa/admission-webhook/deploy/gmsa-webhook.yml.tpl

Lines 105 to 113 in b1f80fc

	volumes:
	- name: tls
	secret:
	secretName: ${NAME}
	items:
	- key: tls_private_key
	path: key
	- key: tls_certificate
	path: crt

but we don't listen for changes, we load it once and start the webhook:

windows-gmsa/admission-webhook/webhook.go

Lines 77 to 82 in b1f80fc

	if tlsConfig == nil {
	err = webhook.server.Serve(keepAliveListener)
	} else {
	err = webhook.server.ServeTLS(keepAliveListener, tlsConfig.crtPath, tlsConfig.keyPath)
	}

This could be updated to watch the file and re-loaded it via TLSConfig.GetCertificate https://pkg.go.dev/net/http#Server.ListenAndServeTLS

[jsturtevant]

The strategy of doing a rolling upgrade is probably just as simple

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jsturtevant]

/remove-lifecycle stale