ESQL: Missing enrich policies on skip_unavailable=true clusters no longer fail the query

[quux00]

Missing enrich policies (or failures while looking up the policies on remote clusters) are no longer
fatal errors for skip_unavailable=true clusters. Those clusters will simply not be included
in the rest of the query.

Partially addresses #114531

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

Hi @quux00, I've created a changelog YAML for you.

[quux00]

Note that mismatches across policies (being checked in the section below here) are still fatal errors for skip_unavailable=true clusters. I started down the road of having these be skippable errors, but that looks rather tricky to pull off. At a minimum, you'd have to partition the policies by "skip_unavailable" and build a canonical list of fields/types/etc. from skip_un=false clusters and then compare the skip_un=true clusters and then if any mismatches are found, those aren't fatal, but you pull that cluster out of the list to be resolved for field-caps. Not impossible but this section would require a significant rewrite, so I decided to only handle missing enrich policies (and policies that have errors on the remote cluster during resolution), but still fail them based on mismatches between policies.

[quux00]

I'm not convinced this block (under (if failures.isEmpty)) will ever execute now, but I was apprehensive to remove it. I couldn't find a way to enter this block based on my testing.

[dnhatn]

@quux00 Can you explain why we choose to treat a missing enrich policy as unavailable? I think we should either fail the query in this case or return a partial result if partial_results is specified. This approach would allow users to fix the query or clusters accordingly.

[quux00]

Can you explain why we choose to treat a missing enrich policy as unavailable?

The definition of skip_unavailable that we are using means that if an error occurs on a remote cluster with skip_unavailable=true, then that should not be a fatal error that fails the query. Instead, we return partial data from other clusters. Since the missing enrich policy will only affect that one remote cluster, it can be safely left out of the query and marked as SKIPPED with a failure notice in the ccs_metadata of the response.

So a missing enrich policy on a remote cluster is a fatal error only if the cluster is skip_unavailable=false (or if it is the local cluster).

return a partial result if partial_results is specified

I'm not sure what you are referring to. At present ES|QL does not support partial results, except for the skip_unavailable handling we've been adding. Marking a cluster as skip_unavailable=true is effectively saying you want partial data from the query, just at a cluster level, not a shard/node level.

[dnhatn]

I think a missing enrich policy error is different from an unavailable one. Could we narrow the scope of this PR to focus only on unavailable nodes or connections?

[smalyshev]

I think generally our assumption has been that virtually any error on a cluster that is marked as skip_unavailable is going to lead us to ignore this cluster, but not fail the whole request - provided the request can be performed at all ignoring that cluster (e.g. if the request only contained data from that cluster, it'll still fail). That includes missing indexes and missing policies too. If that assumption is not correct we need to re-sync and define the behavior we want, but that was also the underlying assumption for #112886

[dnhatn]

See #33915 and #27182 (comment)

[smalyshev]

Yeah it looks like there was the same kind of discussion on _search side, and while the "unavailable" thing is indeed quite misleading, I think the end resolution has been that this option works as "ignore the errors that is possible to ignore" rather than "only ignore errors that have to do with network connections". If I understand this correctly, then it makes sense to follow the same road with ES|QL? It may be a bit confusing that "unavailable" works this way, but IMHO it'd be much more confusing if it worked differently for different types of search. In any case, if there are concerns about it then we probably should raise it on the PM level to ensure we don't misunderstand what is supposed to happen?

[quux00]

Could we narrow the scope of this PR to focus only on unavailable nodes or connections?

That is already done: #115266
as well as handling missing indices errors based on the skip_unavailable setting: #116348

The two in progress PRs around skip_unavailable are to handling missing enrich policies (this PR) and arbitrary errors at execution time (#116365). This will make ES|QL consistent with current skip_unavailable handling in _search CCS.

[dnhatn]

I think we should avoid the mistake we made with the _search API. This flag is inconsistent in _search, depending on whether ccs_minimized_round_trip is enabled. We can't easily fix this because of BWC.

[smalyshev]

But for ES|QL MRT is always on, so there's no room for such inconsistency?

[quux00]

Closing this as we will not be extending skip_unavailable=true to handle this use case.