You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From discussing with @dimitarvdimitrov and @colega, it would seem as if the store-gateway doesn't respect context cancellation while loading index headers. Meaning that even if the query is cancelled, the store-gateway will continue until it's finished loading the requested index headers.
We should investigate if this is actually the case, and consider respecting context cancellation during index header loading.
aknuds1
changed the title
Respect context cancellation when loading index headers
Respect context cancellation while loading index headers
May 25, 2023
we've spent some time to improve lazy loading speed with #5046; also reducing the impact of lazy loading around restarts with #4762 and not blocking queries that have already loaded index headers with #5507
this should make lazy loading have less of an impact on the query path
I'm also not sure whether we should be cancelling the lazy loading process. This way if the query comes back for the same time range, we're effectively throwing away the work done for loading some parts of the index header. I think it would make sense to continue loading the index header. Perhaps retain it with a much smaller TTL and unload it sooner than other blocks, but abroting seems like a waste of work.
In light of @dimitarvdimitrov's comment (also having conferred with him), we can close this issue, as we might not want to cancel the lazy loading process after all. We could still reopen this issue in the future, if we change our mind.
From discussing with @dimitarvdimitrov and @colega, it would seem as if the store-gateway doesn't respect context cancellation while loading index headers. Meaning that even if the query is cancelled, the store-gateway will continue until it's finished loading the requested index headers.
We should investigate if this is actually the case, and consider respecting context cancellation during index header loading.
See also prometheus/prometheus#12359, which may be related(?).
According to @dimitarvdimitrov, we should look for context cancellation handling in this code.
The text was updated successfully, but these errors were encountered: