Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thanos-query does not query data after running for a period of time, health check is ok, but query api timeout. #4948

Closed
DemonLee0719 opened this issue Dec 14, 2021 · 2 comments

Comments

@DemonLee0719
Copy link

Thanos, Prometheus and Golang version used:
thanos: 0.23.1
prometheus: 2.31.1
openshift: v3.11.88
k8s: v1.11.0

Object Storage Provider: None

What happened:
when i curl http://thanos-query-pod-cluster-ip:9090/-/healthy return ok,
but curl http://thanos-query-pod-cluster-ip:9090/api/v1/query?query=up timeout.
Only one of the three pods will have this situation.
Something must be happening inside this pod, but the log can't see anything.
The configuration file is as follows
BE5F69CFD1C6D676DF4BE0DD70A941E8_IMAGE#1639447710769#jpg

B2C4556DAEE1D5188017714AE70843E7_IMAGE#1639447710769#jpg
820847D5DF9D14E85EDB011DCB7646C1_IMAGE#1639447710770#jpg
930989E1D572B67CB3DFAB85EA6B4543_IMAGE#1639447710770#jpg
B8DBA10D4EEC7BE29B019F44F06A4784_IMAGE#1639447710769#jpg
B81D8101C6D67A3D6E5F1384BD6F4311_IMAGE#1639447710770#jpg
BAEE4F062BA91A4175D340C69EF04EF9_IMAGE#1639447710771#jpg

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

@DemonLee0719
Copy link
Author

livenessProbe url modified to /api/v1/query?query=up. When thanos-query does not work, the pod will be restarted.this is my solution, But not the best solution.

@hanjm
Copy link
Member

hanjm commented Dec 14, 2021

same issue with #4766. please use 0.23.2 rc https://github.com/thanos-io/thanos/releases/tag/v0.23.2-rc.0 or 0.24.0 rc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants