Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect context cancellation while loading index headers #5079

Closed
aknuds1 opened this issue May 25, 2023 · 2 comments
Closed

Respect context cancellation while loading index headers #5079

aknuds1 opened this issue May 25, 2023 · 2 comments
Assignees
Labels

Comments

@aknuds1
Copy link
Contributor

aknuds1 commented May 25, 2023

From discussing with @dimitarvdimitrov and @colega, it would seem as if the store-gateway doesn't respect context cancellation while loading index headers. Meaning that even if the query is cancelled, the store-gateway will continue until it's finished loading the requested index headers.

We should investigate if this is actually the case, and consider respecting context cancellation during index header loading.

See also prometheus/prometheus#12359, which may be related(?).

According to @dimitarvdimitrov, we should look for context cancellation handling in this code.

@aknuds1 aknuds1 self-assigned this May 25, 2023
@aknuds1 aknuds1 changed the title Respect context cancellation when loading index headers Respect context cancellation while loading index headers May 25, 2023
@dimitarvdimitrov
Copy link
Contributor

we've spent some time to improve lazy loading speed with #5046; also reducing the impact of lazy loading around restarts with #4762 and not blocking queries that have already loaded index headers with #5507

this should make lazy loading have less of an impact on the query path


I'm also not sure whether we should be cancelling the lazy loading process. This way if the query comes back for the same time range, we're effectively throwing away the work done for loading some parts of the index header. I think it would make sense to continue loading the index header. Perhaps retain it with a much smaller TTL and unload it sooner than other blocks, but abroting seems like a waste of work.

@aknuds1
Copy link
Contributor Author

aknuds1 commented Nov 1, 2023

In light of @dimitarvdimitrov's comment (also having conferred with him), we can close this issue, as we might not want to cancel the lazy loading process after all. We could still reopen this issue in the future, if we change our mind.

@aknuds1 aknuds1 closed this as completed Nov 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants