spicedb-operator does not detect TLS cert change even though it is already rotated

[AnhQuanTrl]

Hi everyone,
Currently I have been facing a strange issue with spicedb-operator. The individual spicedb pod is throwing this error

transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-01-13T04:11:49Z is after 2025-01-10T10:30:15Z

with the CheckPermission API.

I was able to figure out that the certificate that the SpiceDb pod was using was outdated and not the one currently stored in TlsSecertName (this is configured in SpiceDbCluster CRD). I tried to delete all the SpiceDb pods for them to pick up the new certs and it worked!!!

I think spicedb-operator should have a mechanism in place to detect the cert changes and try to restart the SpiceDb pods to prevent certification expiration issue. Otherwise, we will have to perform these manual restart which might be not ideal.

For more information, I am currently following the Self signed certificate example in this repository.

[ecordell]

Thanks for the report @AnhQuanTrl -

SpiceDB should detect cert changes on its own and reload them. I suspect you're actually hitting authzed/spicedb#1448, where the CA certs have rotated, which SpiceDB doesn't currently automatically detect. Are you able to confirm it was only the leaf cert that rotated and not the CA?