Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Frozen tier exclusion doesn't prevent the search API to return failures for missing shards in the frozen tier #114910

Closed
andreidan opened this issue Oct 16, 2024 · 1 comment · Fixed by #114990
Closed

Frozen tier exclusion doesn't prevent the search API to return failures for missing shards in the frozen tier #114910

andreidan opened this issue Oct 16, 2024 · 1 comment · Fixed by #114990
Assignees
@andreidan
Labels
>bug :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@andreidan
Copy link
Contributor

andreidan commented Oct 16, 2024 •
edited
Loading

Elasticsearch Version

8.15, 8.16

Installed Plugins

No response

Java Version

bundled

OS Version

darwin

Problem Description

The _search API still reports failures even if the
data_frozen tier is excluded from the query.
E.g. this query:

POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}

will return

  "_shards": {
    "total": 21,
    "successful": 7,
    "skipped": 0,
    "failed": 14,
    "failures": [
      {
        "shard": 0,
        "index": "partial-.ds-testing-2024.10.11-000001",
        "node": null,
        "reason": {
          "type": "no_shard_available_action_exception",
          "reason": null
        }
      },
      {
        "shard": 0,
        "index": "partial-.ds-testing-2024.10.11-000002",
        "node": null,
        "reason": {
          "type": "no_shard_available_action_exception",
          "reason": null
        }
      }
   ...

We'd expect no partial-* indices to be reported as failures as we're
not interested in searching the frozen tier.

This is due to the can_match phase not skipping the frozen tier
based on the _tier filter.

Steps to Reproduce

Have an Elasticsearch cluster with a node with roles [master, data_content, data_hot] and another node with [data_frozen].
Mount some partially mounted indices (follow https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html )
Stop the frozen node.
Execute a query in the form of

POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}

Logs (if relevant)

No response
@andreidan andreidan added :Search Foundations/Search Catch all for Search Foundations >bug labels Oct 16, 2024
@andreidan andreidan self-assigned this Oct 16, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Oct 16, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@piergm piergm mentioned this issue Oct 18, 2024
Merged
@andreidan andreidan mentioned this issue Oct 18, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue Oct 24, 2024
@andreidan
Allow for queries on _tier to skip shards during coordinator rewrite (#…
327f232 
…114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes #114910
andreidan added a commit to andreidan/elasticsearch that referenced this issue Oct 24, 2024
@andreidan
Allow for queries on _tier to skip shards during coordinator rewrite (e…
ee67f08 
…lastic#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes elastic#114910
andreidan added a commit to andreidan/elasticsearch that referenced this issue Oct 24, 2024
@andreidan
Allow for queries on _tier to skip shards during coordinator rewrite (e…
1aee634 
…lastic#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes elastic#114910
elasticsearchmachine pushed a commit that referenced this issue Oct 24, 2024
@andreidan
[8.16] Allow for queries on _tier to skip shards during coordinator r…
e8877d6 
…ewrite (#114990) (#115513)

* Allow for queries on _tier to skip shards during coordinator rewrite (#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes #114910

* Don't use getFirst

* Test compile
davidkyle pushed a commit to davidkyle/elasticsearch that referenced this issue Oct 24, 2024
@andreidan @davidkyle
Allow for queries on _tier to skip shards during coordinator rewrite (e…
49a5aef 
…lastic#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes elastic#114910
elasticsearchmachine pushed a commit that referenced this issue Oct 24, 2024
@andreidan @elasticmachine
[8.x] Allow for queries on _tier to skip shards during coordinator re…
8a6d692 
…write (#114990) (#115514)

* Allow for queries on _tier to skip shards during coordinator rewrite (#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes #114910

* Don't use getFirst

* test compilation

---------

Co-authored-by: Elastic Machine <[email protected]>
georgewallace pushed a commit to georgewallace/elasticsearch that referenced this issue Oct 25, 2024
@andreidan @georgewallace
Allow for queries on _tier to skip shards during coordinator rewrite (e…
51a7281 
…lastic#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes elastic#114910
@andreidan andreidan mentioned this issue Oct 30, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue Nov 1, 2024
@andreidan
Enable _tier based coordinator rewrites for all indices (not just mou…
71dfb06 
…nted indices) (#115797)

As part of #114990 we
enabled using the `_tier` field as part of the coordinator rewrite in
order to skip  shards that do not match a `_tier` filter, but only for 
fully/partially mounted indices. 

This PR enhances the previous work by allowing a coordinator rewrite to
skip shards that will not match the `_tier` query for all indices
(irrespective of their lifecycle state i.e. hot and warm  indices can
now skip shards based on the `_tier` query)

Note however, that hot/warm indices will not automatically take
advantage of the `can_match` coordinator rewrite  (like read only
indices do) but only the search requests that surpass the
`pre_filter_shard_size` threshold will.

Relates to
[#114910](#114910)
andreidan added a commit to andreidan/elasticsearch that referenced this issue Nov 1, 2024
@andreidan
Enable _tier based coordinator rewrites for all indices (not just mou…
f25b78f 
…nted indices) (elastic#115797)

As part of elastic#114990 we
enabled using the `_tier` field as part of the coordinator rewrite in
order to skip  shards that do not match a `_tier` filter, but only for
fully/partially mounted indices.

This PR enhances the previous work by allowing a coordinator rewrite to
skip shards that will not match the `_tier` query for all indices
(irrespective of their lifecycle state i.e. hot and warm  indices can
now skip shards based on the `_tier` query)

Note however, that hot/warm indices will not automatically take
advantage of the `can_match` coordinator rewrite  (like read only
indices do) but only the search requests that surpass the
`pre_filter_shard_size` threshold will.

Relates to
[elastic#114910](elastic#114910)

(cherry picked from commit 71dfb06)
Signed-off-by: Andrei Dan <[email protected]>
@andreidan andreidan mentioned this issue Nov 1, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue Nov 1, 2024
@andreidan @elasticmachine
[8.x] Enable _tier based coordinator rewrites for all indices (not ju…
d0eb5a0 
…st mounted indices) (#115797) (#116076)

* Enable _tier based coordinator rewrites for all indices (not just mounted indices) (#115797)

As part of #114990 we
enabled using the `_tier` field as part of the coordinator rewrite in
order to skip  shards that do not match a `_tier` filter, but only for
fully/partially mounted indices.

This PR enhances the previous work by allowing a coordinator rewrite to
skip shards that will not match the `_tier` query for all indices
(irrespective of their lifecycle state i.e. hot and warm  indices can
now skip shards based on the `_tier` query)

Note however, that hot/warm indices will not automatically take
advantage of the `can_match` coordinator rewrite  (like read only
indices do) but only the search requests that surpass the
`pre_filter_shard_size` threshold will.

Relates to
[#114910](#114910)

(cherry picked from commit 71dfb06)
Signed-off-by: Andrei Dan <[email protected]>

* Fix test compilation

---------

Co-authored-by: Elastic Machine <[email protected]>
jfreden pushed a commit to jfreden/elasticsearch that referenced this issue Nov 4, 2024
@andreidan @jfreden
Allow for queries on _tier to skip shards during coordinator rewrite (e…
5b9342a 
…lastic#114990)

The `_tier` metadata field was not used on the  coordinator when
rewriting queries in order to exclude shards that don't match. This lead
to queries in the following form to continue to report failures even
though the only unavailable shards were in the  tier that was excluded
from search (frozen tier in this example):

```
POST testing/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "term": {
            "_tier": "data_frozen"
          }
        }
      ]
    }
  }
}
```

This PR addresses this by having the queries that can execute on `_tier`
(term, match, query string, simple query string, prefix, wildcard)
execute a coordinator rewrite to  exclude the indices that don't match
the `_tier` query  **before** attempting to reach to the shards (shards,
that might not be available and raise errors). 

Fixes elastic#114910
jfreden pushed a commit to jfreden/elasticsearch that referenced this issue Nov 4, 2024
@andreidan @jfreden
Enable _tier based coordinator rewrites for all indices (not just mou…
87ad2be 
…nted indices) (elastic#115797)

As part of elastic#114990 we
enabled using the `_tier` field as part of the coordinator rewrite in
order to skip  shards that do not match a `_tier` filter, but only for 
fully/partially mounted indices. 

This PR enhances the previous work by allowing a coordinator rewrite to
skip shards that will not match the `_tier` query for all indices
(irrespective of their lifecycle state i.e. hot and warm  indices can
now skip shards based on the `_tier` query)

Note however, that hot/warm indices will not automatically take
advantage of the `can_match` coordinator rewrite  (like read only
indices do) but only the search requests that surpass the
`pre_filter_shard_size` threshold will.

Relates to
[elastic#114910](elastic#114910)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andreidan andreidan

Labels
>bug :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow for queries on _tier to skip shards during coordinator rewrite andreidan/elasticsearch
2 participants
@andreidan @elasticsearchmachine