consul builtin sidecar proxy takes 10 minutes to start working #12319
Assignees
Labels
theme/connect
Anything related to Consul Connect, Service Mesh, Side Car Proxies
theme/performance
Performance benchmarking or potential improvement
type/bug
Feature does not function as expected
Comments
Amier3
added
type/bug
|
Hey @jrnijboer Thanks for bringing this to our attention. After looking at the changelog we think we have a good idea of what changes would cause this in 1.9.14, but it'll take a deeper dive to be sure. We'll keep this issue updated with what we find. |
|
After some investigation, looks like this may be the result of #11693 which caused a regression: #12048. Basically, a non-blocking query which follows a blocking query on a leaf cert will block, meaning we do not see longs until the timeout (10min) is reached. 1.9.13: 1.9.14 Note the We are still investigating a fix for this issue. Thank you for your patience! |
kisunji
removed
the
needs-investigation
|
Is there any update on this issue? We have currently put a hold on updating our Consul installations untill this issue is fixed |
rboyer
added a commit
that referenced
this issue
Apr 20, 2022
…king leaf cert query to block (#12820) Fixes #12048 Fixes #12319 Regression introduced in #11693 Local reproduction steps: 1. `consul agent -dev` 2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'` 3. make note of the `X-Consul-Index` header returned 4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'` 5. Kill the above curl when it hangs with Ctrl-C 6. Repeat (2) and it should not hang.
hc-github-team-consul-core
pushed a commit
that referenced
this issue
Apr 20, 2022
…king leaf cert query to block (#12820) Fixes #12048 Fixes #12319 Regression introduced in #11693 Local reproduction steps: 1. `consul agent -dev` 2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'` 3. make note of the `X-Consul-Index` header returned 4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'` 5. Kill the above curl when it hangs with Ctrl-C 6. Repeat (2) and it should not hang.
hc-github-team-consul-core
pushed a commit
that referenced
this issue
Apr 20, 2022
…king leaf cert query to block (#12820) Fixes #12048 Fixes #12319 Regression introduced in #11693 Local reproduction steps: 1. `consul agent -dev` 2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'` 3. make note of the `X-Consul-Index` header returned 4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'` 5. Kill the above curl when it hangs with Ctrl-C 6. Repeat (2) and it should not hang.
hc-github-team-consul-core
pushed a commit
that referenced
this issue
Apr 20, 2022
…king leaf cert query to block (#12820) Fixes #12048 Fixes #12319 Regression introduced in #11693 Local reproduction steps: 1. `consul agent -dev` 2. `curl -sLiv 'localhost:8500/v1/agent/connect/ca/leaf/web'` 3. make note of the `X-Consul-Index` header returned 4. `curl -sLi 'localhost:8500/v1/agent/connect/ca/leaf/web?index=<VALUE_FROM_STEP_3>'` 5. Kill the above curl when it hangs with Ctrl-C 6. Repeat (2) and it should not hang.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Overview of the Issue
When starting the builtin sidecar proxy to enable Consul Connect for a service registered in Consul, the proxy process starts immediately but only after 10 minutes it starts working and accepting Consul Connect requests. This used to work immediately in older versions of Consul. I've found out that Consul version 1.9.13 seems to work and in version 1.9.14 it takes 10 minutes. The release notes don't hint at any changes to the Consul connect proxy though. I have not done an exhaustive search to check if this happens with all versions from 1.9.4 and later or that all versions from 1.9.3 and earlier do not have this bug.
In our dev-stage we have consul servers running on version 1.11.1. On that dev-stage it is possible to use the sidecar proxy when using Consul 1.9.13 on a client. I suppose this indicates that it's not a server problem?
Reproduction Steps
Steps to reproduce this issue:
Download the latest consul version (1.11.2 at this moment)
Now the sidecar proxy should start and open a socket on port 21000. For consul versions 1.9.13 and older this works nearly instant, for newer versions it takes 10 minutes to work. The first message in the log that appears is
[INFO] proxy: Proxy loaded config and ready to serveThis is immediately followed by:
Consul info for both Client and Server
This happens on a consul cluster in dev modus, as well as a multi-node consul cluster setup with ACL's and TLS.
Operating system and Environment details
This is reproducible on Linux (Ubuntu 20.04) as well as Windows 10 and WSL.
Log Fragments
logs from the consul agent:
=======================
logs from the sidecar proxy (please not the date output and the timestamps of the logs):
The text was updated successfully, but these errors were encountered: