Skip to content

Commit

Permalink
vault: avoid continual renewal of invalid token (#18985)
Browse files Browse the repository at this point in the history
A series of errors may happen when a token is invalidated while the
Vault client is waiting to renew it. The token may have been invalidated
for several reasons, such as the alloc finished running and it's now
terminal or the token may have been change directly on Vault
out-of-band.

Most of the errors are caused by retries that will never succeed until
Vault fully removes the token from its state.

This commit prevents the retries by making the error `invalid lease ID`
a fatal error.

In earlier versions of Vault, this case was covered by the error `lease
not found or lease is not renewable`, which is already considered to be
a fatal error by Nomad:

https://github.com/hashicorp/vault/blob/2d0cde4ccc0323591d9414342cb15f5cb70271d7/vault/expiration.go#L636-L639

But hashicorp/vault#5346 introduced an earlier
`nil` check that generates a different error message:

https://github.com/hashicorp/vault/blob/750ab337eaa0b049d9cf1535c00e860129e5e9a0/vault/expiration.go#L1362-L1364

Both errors happen for the same reason (`le == nil`) and so should be
considered fatal on renewal.
  • Loading branch information
lgfa29 authored Nov 8, 2023
1 parent 7054fe1 commit ab36cf0
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 0 deletions.
3 changes: 3 additions & 0 deletions .changelog/18985.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:bug
vault: Fixed an issue that could cause Nomad to attempt to renew a Vault token that is already expired
```
1 change: 1 addition & 0 deletions client/vaultclient/vaultclient.go
Original file line number Diff line number Diff line change
Expand Up @@ -512,6 +512,7 @@ func (c *vaultClient) renew(req *vaultClientRenewalRequest) error {
fatal := false
if renewalErr != nil &&
(strings.Contains(renewalErr.Error(), "lease not found or lease is not renewable") ||
strings.Contains(renewalErr.Error(), "invalid lease ID") ||
strings.Contains(renewalErr.Error(), "lease is not renewable") ||
strings.Contains(renewalErr.Error(), "token not found") ||
strings.Contains(renewalErr.Error(), "permission denied")) {
Expand Down

0 comments on commit ab36cf0

Please sign in to comment.