Query: Allow to ignore No StoreAPIs matched error

[junotx]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Add ignore_no_stores_matched param to Query APIs to allow to ignore No StoreAPIs matched error (default to be specified by --query.ignore-no-stores-matched flag). The query param is also to added to grpc series request for ignoring the same error.

It may fix the issue #5862 .

Verification

[fpetkovski]

I am not sure if this is a good idea to apply by default. In rulers we usually want to catch errors like this one. However, having a flag to disable this warning might be something useful.

[GiedriusS]

Mhm, I wonder if this is not some issue introduced during the recent proxy.go refactoring. See #5862. It's probably better to investigate what behaviour has changed between versions instead of adding yet another option :/

[junotx]

Mhm, I wonder if this is not some issue introduced during the recent proxy.go refactoring. See #5862. It's probably better to investigate what behaviour has changed between versions instead of adding yet another option :/

I reproduce the No StoresAPIs matched error by running receive with an ingester mode and a nonexistent datadir and then using querier to query it before it receives any remote write requests. It has no stores (did not create a tsdb store for the default tenant id) and but still receive query requests and send the error back to querier.

[junotx]

If the configured datadir not exists, receive ingester will not create a default tenant data dir and not create tsdb store for the default tenant id when open MultiTSDB, so it have no any stores but has an infinite TimeRange(MinInt64->MaxInt64) and then can receive query requests from querier.

This may be a special one of cases mentioned by 5862#issuecomment-1312296482.

[stale]

Hello 👋 Looks like there was no activity on this amazing PR for the last 30 days.
Do you mind updating us on the status? Is there anything we can help with? If you plan to still work on it, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next week, this issue will be closed (we can always reopen a PR if you get back to this!). Alternatively, use remind command if you wish to be reminded at some point in future.

[SuperQ]

Ping, is this still being worked on?

[fpetkovski]

Possibly related issue: #5230 (comment)

[junotx]

conflicts resolved and changelog added

[SuperQ]

Do we actually need to plumb in a configuration option for this? Based on the discussion so far, this is a simple bug and we can just make the change.

[benjaminhuo]

Do we actually need to plumb in a configuration option for this? Based on the discussion so far, this is a simple bug and we can just make the change.

@SuperQ You mean we can directly return nil by removing https://github.com/thanos-io/thanos/pull/5937/files#diff-fc53ebad7275ac8861ae5e9e4e937a7cb9c6b267812df9a61f661c842ed94800R284-R289 ?

[SuperQ]

I think so. I created #6082 as an alternate proposal.

[benjaminhuo]

I think so. I created #6082 as an alternate proposal.

Make sense, no error should be returned if no store matched

[bwplotka]

Fixed in #6082, hope that makes sense @junotx - if not let us know 🤗 Closing the PR for now, but we can reopen if we have some improvements to the merged solution.