Thanos Ruler fails to evaluate all recording rules correctly

[sharathfeb12]

I am currently running Thanos v0.24.0-rc.0.

Few recording rules are evaluated fine while few recording rules seems to be last evaluated 2 days back. This happens very frequently. Restarting the pod fixes the issue temporarily. This issue is reproducible on v0.23.0 as well.

Here is the args passed to the ruler:

- rule
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --objstore.config=$(OBJSTORE_CONFIG)
- --data-dir=/thanos/data
- --eval-interval=1m
- --label=rule_replica="$(NAME)"
- --alert.label-drop=rule_replica
- --remote-write.config-file=/etc/thanos/conf/rw-config.yaml
- --query=dnssrv+_http._tcp.observatorium-thanos-query.monitoring.svc.cluster.local
- --rule-file=/etc/thanos/rules/*/*.yaml

[GiedriusS]

Hello, could you please dump the goroutine stacks when this happens and upload them?

[bwplotka]

Thanks for reporting! pprof profiles available at /debug/pprof/goroutines done at the moment of things being stuck, would be super helpful!

[jleloup]

Isn't this issue similar to #4772 ?

[jleloup]

I think the .pprof file I uploaded there is actually more relevant in this issue as the behaviour I got in my Thanos Ruler is more comparable to this issue as restarting those pods helped only for some dozens of minutes before failing again to process records.

Link to the .pprof: #4772 (comment)

[ahurtaud]

I have the same issue on v0.24.0
Here the goroutine pprof after long time being stuck:
ruler-goroutine.zip

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗