Configurable Vault secrets renewal attempt interval

[marcelocarlos]

Hi,

We are currently looking into ways of improving our integration between Vault and Applications in kubernetes. A very common pattern is to use consul-template as a sidecar container and let it handle secrets retrieval and renewal, and it works very well!

However, there are two items we would like to further customise. The first is monitoring (which is a very similar use case of #1338). The second, which this ticket is about, is having the ability of tune the secrets renewal interval defined in

consul-template/dependency/vault_common.go

Lines 145 to 164 in 5524082

	if base <= 0 {
	base = int(VaultDefaultLeaseDuration.Seconds())
	}
	// Convert to float seconds.
	sleep := float64(time.Duration(base) * time.Second)
	if vaultSecretRenewable(s) {
	// Renew at 1/3 the remaining lease. This will give us an opportunity to retry
	// at least one more time should the first renewal fail.
	sleep = sleep / 3.0
	// Use some randomness so many clients do not hit Vault simultaneously.
	sleep = sleep * (rand.Float64() + 1) / 2.0
	} else {
	// For non-renewable leases set the renew duration to use much of the secret
	// lease as possible. Use a stagger over 85%-95% of the lease duration so that
	// many clients do not hit Vault simultaneously.
	sleep = sleep * (.85 + rand.Float64()*0.1)
	}

.

Ideally, for renewable secrets we'd like to be able to be able to define when (within the lease validity period) the renew attempts would be performed and how many attempts should be done. For example, in a lease valid for 24h, we would like to have 5 attempts to renew the lease, potentially more towards the end of the lease, similar to what's being done to non-renewable secrets. For non-renewable secrets, we would like to customise the number of attempts and the window used.

Assuming this is possible, we could report the attempts in prometheus format so this can be monitored, alerted and acted upon (assuming #1338 is implemented).

In summary, we could maybe first allow customising the number of attempts (make the current 3 hard-coded attempts a customisable setting for renewable secrets and adding it for non-renewable secrets too). Then also allow specifying when the attempts would take place (allow customising the .85 hard-coded value in the non-renewable secrets and add it for renewable secrets as well).

Thank you!

[eikenb]

Hey @marcelocarlos, thanks for the interest.

We just added support for using rotation_period settings on non-renewable secrets. You can see the changes in #1358, which has been merged (only in master for now). Thought I'd mention it in case you would find that useful.

[phemmer]

I'm also interested in such a feature, but would like to set a different interval for k/v secrets fetched by version, possibly with the ability to set an "infinite duration" interval. The only possibility for a k/v versioned secret to change is through deletion.
In my use case, I guarantee a secret version will never be deleted/destroyed (consul-template has terrible handling of deleted secrets), so there's no need to ever re-check it.

[Oloremo]

Interested in this as well. We need to set renewal time to create some guarantees with our monitoring.