Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Watch of type connect_leaf blocks until timeout on service restart #13302

Closed
iSchluff opened this issue May 31, 2022 · 2 comments
Closed

Watch of type connect_leaf blocks until timeout on service restart #13302

iSchluff opened this issue May 31, 2022 · 2 comments

Comments

@iSchluff
Copy link

Overview of the Issue

On service restart for a connect native service I would expect to be able to re-retrieve the leaf certificate without waiting for the update timeout of 10 minutes. Currently however the connect_leaf watch will always block until the timeout of 10m expires from the last issue.

The service I am trying to use is traefik in connect native mode with the consul catalog provider. The effect is that on same-node restarts the service is unusable for up to 10m on restart.

I would expect the watch to provide the current certificate immediately on restart.

Reproduction Steps

Set up a connect native service, request a leaf cert via watch. Then restart that service, the service will not be issued a certificate until the 10m timeout expires.

Complete minimal example: https://gist.github.com/iSchluff/d18e6afc670eeb2355011578ea90e425
Just the connect_leaf watch code from the traefik provider with example nomad job.

Log

2022/05/31 09:14:20 start handler
2022/05/31 09:14:20 got cert -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt0MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDg0ODA4
WhcNMjIwNjAzMDg0ODA4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyX9Z
QEPF3bDKUJCMyCmiWCHvECMr+eTIfUJZepCXEJndwZ5IdjM9vRgHhFmcTL+bfMTN
qmGuKWpxU6cqRor6q6OB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIFl6g8wV
G0tNZTJJ3wRmXi7Ybfy8H21iBvWD3ThuA0sRMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAzS6V
1+Gpr9Eh3ZoFf8WpiKn0w4Q7LaQBhjcZINa4VkACIQDG2rSPyZ2fD9pLyoZykIvR
vQbhKN8clgRHA2a0na/UPA==
-----END CERTIFICATE-----

### service restart

2022/05/31 09:14:37 start handler
2022/05/31 09:24:20 got cert -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt0MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDg0ODA4
WhcNMjIwNjAzMDg0ODA4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyX9Z
QEPF3bDKUJCMyCmiWCHvECMr+eTIfUJZepCXEJndwZ5IdjM9vRgHhFmcTL+bfMTN
qmGuKWpxU6cqRor6q6OB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIFl6g8wV
G0tNZTJJ3wRmXi7Ybfy8H21iBvWD3ThuA0sRMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAzS6V
1+Gpr9Eh3ZoFf8WpiKn0w4Q7LaQBhjcZINa4VkACIQDG2rSPyZ2fD9pLyoZykIvR
vQbhKN8clgRHA2a0na/UPA==
-----END CERTIFICATE-----

Consul info for both Client and Server

Client info 
agent:
        check_monitors = 0
        check_ttls = 0
        checks = 4
        services = 4
build:
        prerelease =
        revision = 09a8cdb4
        version = 1.12.0
consul:
        acl = enabled
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = 172.20.100.43:8300
        server = true
raft:
        applied_index = 2752612
        commit_index = 2752612
        fsm_pending = 0
        last_contact = 29.953429ms
        last_log_index = 2752612
        last_log_term = 71257
        last_snapshot_index = 2737416
        last_snapshot_term = 71255
        latest_configuration = [{Suffrage:Voter ID:161382c7-d6c1-16d3-e513-31cc25963664 Address:172.20.100.43:8300} {Suffrage:Voter ID:70f4322b-fd2b-9267-4fb8-08ddf5465d99 Address:172.20.100.42:8300} {Suffrage:Voter ID:6467fd18-b25b-9c06-8ef1-3628beff1ff4 Address:172.20.100.41:8300}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 71257
runtime:
        arch = amd64
        cpu_count = 24
        goroutines = 155
        max_procs = 24
        os = linux
        version = go1.18.1
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 123
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 7424
        members = 6
        query_queue = 0
        query_time = 5
serf_wan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 3776
        members = 3
        query_queue = 0
        query_time = 5
Server info 
agent:
        check_monitors = 0
        check_ttls = 0
        checks = 4
        services = 4
build:
        prerelease =
        revision = 09a8cdb4
        version = 1.12.0
consul:
        acl = enabled
        bootstrap = false
        known_datacenters = 1
        leader = false
        leader_addr = 172.20.100.43:8300
        server = true
raft:
        applied_index = 2752612
        commit_index = 2752612
        fsm_pending = 0
        last_contact = 29.953429ms
        last_log_index = 2752612
        last_log_term = 71257
        last_snapshot_index = 2737416
        last_snapshot_term = 71255
        latest_configuration = [{Suffrage:Voter ID:161382c7-d6c1-16d3-e513-31cc25963664 Address:172.20.100.43:8300} {Suffrage:Voter ID:70f4322b-fd2b-9267-4fb8-08ddf5465d99 Address:172.20.100.42:8300} {Suffrage:Voter ID:6467fd18-b25b-9c06-8ef1-3628beff1ff4 Address:172.20.100.41:8300}]
        latest_configuration_index = 0
        num_peers = 2
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Follower
        term = 71257
runtime:
        arch = amd64
        cpu_count = 24
        goroutines = 155
        max_procs = 24
        os = linux
        version = go1.18.1
serf_lan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 123
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 7424
        members = 6
        query_queue = 0
        query_time = 5
serf_wan:
        coordinate_resets = 0
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 3776
        members = 3
        query_queue = 0
        query_time = 5

Operating system and Environment details

Ubuntu 20.04.4 LTS
Linux 5.4.0-113-generic
@iSchluff
Copy link
Author

One could argue that this is just the behaviour of the blocking api, but I can't even request a cert with a non-blocking get before starting the watch as that will block aswell.

https://gist.github.com/iSchluff/0e062b036b6643a47861fba2bd362177

2022/05/31 09:43:38 service started
2022/05/31 09:43:38 get leaf took 24.677199ms
2022/05/31 09:43:38 -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt2MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDk0MjM4
WhcNMjIwNjAzMDk0MjM4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE32Dt
7hLjrT0vTN/FN45Av2VGHdZvpo8ziQFgr+9ClJJszBibs3/5jxqrSFyhzvpXZQRJ
eCH0hC6iyiE9dFlB2aOB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIN+yxUcw
Yt8BEbkUogu5oZIkfwQpFoP35eKhanIPWZnZMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAhd1c
Wed9KopJPQkut36A2o+FlRRXCL3T9M0Rdp5FnxwCIQCtJhlECF2FzfoJyc74UZgF
NgoIBKP+EPqJ54oshpID4A==
-----END CERTIFICATE-----

2022/05/31 09:43:38 start handler
2022/05/31 09:43:38 got cert -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt2MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDk0MjM4
WhcNMjIwNjAzMDk0MjM4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE32Dt
7hLjrT0vTN/FN45Av2VGHdZvpo8ziQFgr+9ClJJszBibs3/5jxqrSFyhzvpXZQRJ
eCH0hC6iyiE9dFlB2aOB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIN+yxUcw
Yt8BEbkUogu5oZIkfwQpFoP35eKhanIPWZnZMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAhd1c
Wed9KopJPQkut36A2o+FlRRXCL3T9M0Rdp5FnxwCIQCtJhlECF2FzfoJyc74UZgF
NgoIBKP+EPqJ54oshpID4A==
-----END CERTIFICATE-----

2022/05/31 09:43:51 service started
2022/05/31 09:53:38 get leaf took 9m46.95118068s
2022/05/31 09:53:38 -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt2MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDk0MjM4
WhcNMjIwNjAzMDk0MjM4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE32Dt
7hLjrT0vTN/FN45Av2VGHdZvpo8ziQFgr+9ClJJszBibs3/5jxqrSFyhzvpXZQRJ
eCH0hC6iyiE9dFlB2aOB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIN+yxUcw
Yt8BEbkUogu5oZIkfwQpFoP35eKhanIPWZnZMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAhd1c
Wed9KopJPQkut36A2o+FlRRXCL3T9M0Rdp5FnxwCIQCtJhlECF2FzfoJyc74UZgF
NgoIBKP+EPqJ54oshpID4A==
-----END CERTIFICATE-----

2022/05/31 09:53:38 start handler
2022/05/31 09:53:38 got cert -----BEGIN CERTIFICATE-----
MIICHDCCAcGgAwIBAgIDDkt2MAoGCCqGSM49BAMCMDAxLjAsBgNVBAMTJXByaS0x
b2g2emtxLmNvbnN1bC5jYS5mY2MyODFmMC5jb25zdWwwHhcNMjIwNTMxMDk0MjM4
WhcNMjIwNjAzMDk0MjM4WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE32Dt
7hLjrT0vTN/FN45Av2VGHdZvpo8ziQFgr+9ClJJszBibs3/5jxqrSFyhzvpXZQRJ
eCH0hC6iyiE9dFlB2aOB+TCB9jAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYI
KwYBBQUHAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIN+yxUcw
Yt8BEbkUogu5oZIkfwQpFoP35eKhanIPWZnZMCsGA1UdIwQkMCKAIGouTrCrjfHe
svy/XuKKH9bOu98sD99c0WMQh4724/FqMF8GA1UdEQEB/wRVMFOGUXNwaWZmZTov
L2ZjYzI4MWYwLWUzNTEtNmEyOS0zYjc4LWMyMDdjNjlmYWE0Ny5jb25zdWwvbnMv
ZGVmYXVsdC9kYy9kYzEvc3ZjL2xlYWZlcjAKBggqhkjOPQQDAgNJADBGAiEAhd1c
Wed9KopJPQkut36A2o+FlRRXCL3T9M0Rdp5FnxwCIQCtJhlECF2FzfoJyc74UZgF
NgoIBKP+EPqJ54oshpID4A==
-----END CERTIFICATE-----

Basically it seems like the blocking query prevents any chance of getting a new cert without waiting until 10m are over.

@iSchluff
Copy link
Author

iSchluff commented Jun 1, 2022

Should be fixed by #12820 I guess

@iSchluff iSchluff closed this as completed Jun 1, 2022
johnalotoski added a commit to input-output-hk/bitte that referenced this issue Aug 24, 2022
@johnalotoski
bump: consul 1.11.2 --> 1.13.1
bec9d7a 
* Fixes traefik slow restart issue: hashicorp/consul#13302
johnalotoski added a commit to input-output-hk/bitte that referenced this issue Aug 29, 2022
@johnalotoski
bump: consul 1.11.2 --> 1.13.1
9b2483a 
* Fixes traefik slow restart issue: hashicorp/consul#13302
johnalotoski added a commit to input-output-hk/bitte that referenced this issue Aug 31, 2022
@johnalotoski
bump: consul 1.11.2 --> 1.13.1
bc4374c 
* Fixes traefik slow restart issue: hashicorp/consul#13302
* Updates the idle_timeout Consul patch for the new version bump code
johnalotoski added a commit to input-output-hk/bitte that referenced this issue Sep 15, 2022
@johnalotoski
bump: consul 1.11.2 --> 1.13.1
d1885c0 
* Fixes traefik slow restart issue: hashicorp/consul#13302
* Updates the idle_timeout Consul patch for the new version bump code
johnalotoski added a commit to input-output-hk/bitte that referenced this issue Sep 22, 2022
@johnalotoski
bump: consul 1.11.2 --> 1.13.1
bf6f75d 
* Fixes traefik slow restart issue: hashicorp/consul#13302
* Updates the idle_timeout Consul patch for the new version bump code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@iSchluff