Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consul template doesn't renew Vault token and quits after 12 attempts #1161

Closed
gerases opened this issue Dec 9, 2018 · 7 comments
Closed

Consul template doesn't renew Vault token and quits after 12 attempts #1161

gerases opened this issue Dec 9, 2018 · 7 comments
Labels
bug

Comments

@gerases
Copy link

gerases commented Dec 9, 2018 •
edited
Loading

Consul Template version

consul-template v0.19.5 (57b6c71)

Configuration

hcl
consul {
  token = "SANITIZED"
  address = "SANITIZED"
}

vault {
  address = "SANITIZED"
  grace = "5m"
  token = 'SANITIZED'
  unwrap_token = false
  renew_token = true
  retry {
    enabled = true
  }
}

{{ with secret "SANITIZED" -}}
{{ .Data.data.secret }}
{{ end -}}

Command

./consul-template -log-level trace -config cfg.hcl -template "in.tpl:out.txt"

Debug output

https://gist.github.com/gerases/0b272ce919f7dc4f20d08376b3f99f55

Expected behavior

I issued a token with a TTL of 10 minutes. I thought consul-template would refresh the Vault token. Refreshing the token manually does work and the token is renewable.

The trace messages are a bit confusing:

2018/12/08 21:13:40.539371 [TRACE] (view) vault.token starting fetch
2018/12/08 21:13:40.539406 [TRACE] vault.token: starting renewer
2018/12/08 21:13:40.545724 [TRACE] vault.token: successfully renewed
2018/12/08 21:13:40.545742 [WARN] vault.token: renewer returned (maybe the lease expired)
2018/12/08 21:13:40.545762 [TRACE] vault.token: token is not renewable, sleeping for 5m0s

Was the token renewed or not? Looking at the token properties, I see that it gets renewed by ~2 mins each time.

Actual behavior

Consul-template quits after 12 attempts though the token does seem to get refreshed by ~2 mins each of the 12 times.
@gerases
Copy link
Author

gerases commented Dec 12, 2018

Reading around, I now realize that the "lease" consul-template is talking about is about leases related to dynamic secrets. In my case, the secret itself is static. I don't need to renew its lease. I just need consul-template to refresh the connection token.

@gerases
Copy link
Author

gerases commented Dec 27, 2018

Anybody?

@a-nldisr
Copy link

Would love to have an update on this too!

@primeroz
Copy link

I am hitting this as well with 0.19.5

In my case i can see the token being renewed but the error, and the backoff, is still being printed

vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:33:43.247083 [WARN] (view) lease expired or is not renewable (retry attempt 1 after "250ms")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:33:43.652649 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:38:43.652825 [WARN] (view) lease expired or is not renewable (retry attempt 2 after "500ms")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:38:44.328582 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:43:44.329049 [WARN] (view) lease expired or is not renewable (retry attempt 3 after "1s")  
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:43:45.522034 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:48:45.522263 [WARN] (view) lease expired or is not renewable (retry attempt 4 after "2s")   
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:48:47.719767 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:53:47.720045 [WARN] (view) lease expired or is not renewable (retry attempt 5 after "4s")   
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:53:52.051683 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:58:52.052398 [WARN] (view) lease expired or is not renewable (retry attempt 6 after "8s")   
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 18:59:00.204343 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:04:00.204457 [WARN] (view) lease expired or is not renewable (retry attempt 7 after "16s")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:04:16.401974 [WARN] vault.token: renewer returned (maybe the lease expired)          
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:09:16.408304 [WARN] (view) lease expired or is not renewable (retry attempt 8 after "32s")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:09:48.671377 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:14:48.671672 [WARN] (view) lease expired or is not renewable (retry attempt 9 after "1m0s")                                                                                  
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:15:48.913092 [WARN] vault.token: renewer returned (maybe the lease expired)                                                                                                  
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:20:48.913571 [WARN] (view) lease expired or is not renewable (retry attempt 10 after "1m0s")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:21:54.233862 [WARN] vault.token: renewer returned (maybe the lease expired)                                                                                                  
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:26:54.234157 [WARN] (view) lease expired or is not renewable (retry attempt 11 after "1m0s")                                                                                 
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:27:54.422294 [WARN] vault.token: renewer returned (maybe the lease expired)                                                                                                  
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:32:54.422634 [WARN] (view) lease expired or is not renewable (retry attempt 12 after "1m0s")                                                                                 
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:33:54.812902 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:38:54.813279 [ERR] (view) lease expired or is not renewable (exceeded maximum retries)                                                                                       
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:38:54.813344 [ERR] (runner) watcher reported error: lease expired or is not renewable
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:38:54.813364 [ERR] (cli) lease expired or is not renewable
                                                                                                                       
^C                                                                                                                                                                                                                                            
 przx1  sidecar  stern -n default -l app=vault-client-demo                                                           
+ vault-client-demo-585d8d48cc-hbqlp › busybox                                                                         
+ vault-client-demo-585d8d48cc-hbqlp › gostatic                                                                        
+ vault-client-demo-585d8d48cc-hbqlp › consul-template
vault-client-demo-585d8d48cc-hbqlp gostatic 2019/05/29 18:28:48 Listening at 0.0.0.0:8043 /...
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:38:55.920949 [WARN] (clients) disabling vault SSL verification
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:38:56.321203 [WARN] vault.token: renewer returned (maybe the lease expired)
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:43:56.321509 [WARN] (view) lease expired or is not renewable (retry attempt 1 after "250ms")
vault-client-demo-585d8d48cc-hbqlp consul-template 2019/05/29 19:43:56.834578 [WARN] vault.token: renewer returned (maybe the lease expired)

After the 12th error consul-template dies and kubernetes restart it

@eikenb eikenb added the bug label Jun 14, 2019
@eikenb
Copy link
Contributor

eikenb commented Aug 6, 2019

Seems like this and #1224 are related.

@catsby
Copy link
Contributor

catsby commented Sep 19, 2019

Hey @eikenb - do you think this is fixed with #1224 by #1269 ?

@eikenb
Copy link
Contributor

eikenb commented Sep 20, 2019

Thanks @catsby, this is indeed fixed.

@eikenb eikenb closed this as completed Sep 20, 2019
@szymonpk szymonpk mentioned this issue Apr 7, 2020
Merged
szymonpk added a commit to szymonpk/bank-vaults that referenced this issue Apr 7, 2020
@szymonpk
update consul-template default image
707c3e1 
Update the consul-image to the 0.24.1-alpine. In the previous versions,
there was a bug that caused issues with token renewal. Which could
result in a pod/container crash loop.

Reference:
hashicorp/consul-template#1161
bonifaido pushed a commit to bank-vaults/bank-vaults that referenced this issue Apr 7, 2020
@szymonpk
update consul-template default image (#935)
d4b68c8 
Update the consul-image to the 0.24.1-alpine. In the previous versions,
there was a bug that caused issues with token renewal. Which could
result in a pod/container crash loop.

Reference:
hashicorp/consul-template#1161
@Ardiea Ardiea mentioned this issue Mar 10, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@catsby @eikenb @a-nldisr @primeroz @gerases