#4657 broke query -> sidecar communication

[sylr]

Thanos, Prometheus and Golang version used:

Thanos: HEAD
Go: 1.17.1

Object Storage Provider:

S3

What happened:

While testing #4667 I merged the main branch into it locally and it broke the communication between my thanos query and thanos sidecar.

I bisected and reverting #4657 resolved the issue.

I don't know if it is relevant but I have a traefik reverse proxy between thanos-query and the incriminated thanos-sidecar.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Thanos Query Logs

level=warn ts=2021-09-21T08:39:46.659306271Z caller=endpointset.go:511 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090
level=warn ts=2021-09-21T08:39:51.660095416Z caller=endpointset.go:511 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090
level=warn ts=2021-09-21T08:39:56.660884651Z caller=endpointset.go:511 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090

Anything else we need to know:

[hanjm]

address=prometheus-thanos-grpc-eks-euc1-02.dev.company.com:9090

9090 is the Prometheus port, not the thanos-sidecar port.
Could you provide the thanos-query and thanos-sidecar command args?

[sylr]

Ok so I don't reproduce anymore ...

I'm sorry for the annoyance.

[GiedriusS]

It's ok 👍 no worries 🤗

[hanjm]

if the port 9090 is point to sidecar correctly. I think the error log is produced when sidecar is up but prometheus is not ready，so query timeout by context deadline exceeded.

[sylr]

9090 was the port on the reverse proxy which was forwarding on 10901. The mistery lies elsewhere ...

Sorry again to have wasted your time.

[bwplotka]

No worries, good we found the issue and it's not a release blocker! 🤗