Watcher: Ensure upgrading from 6.x does not break watches checking for hit count

[spinscale]

Many watches check the hit count in their condition (either a script condition or a compare condition) in order to determine if any documents have been returned.

As Elasticsearch 7.0 will not return hit counts by default, those watches would break silently on upgrade. We need to come up with a strategy to keep those watches running and/or properly notify the user that a new parameter is required.

A couple of common strategies:

1. Always add the correct header to the search to return hit counts. Then no watch ever will profit from the speed up we have gained by disabling hit counts. This is fully BWC compatible
2. Use the Upgrade API to check the conditions (the main condition plus all the action conditions) if there is a mention of hits.total, and if that is the case, change the search inputs to include that parameter. Then we can run the upgrade API before upgrading and thus change each watch accordingly. Note that this means, checking all the nested actions in chained actions as well.

Also, it might make sense to spill a useful error message, if a user is missing this parameter but specifies hits.total in a watch, to ease the switch from 6.0 to 7.0 - otherwise the user will only notice this new behaviour when checking the watch history or running the execute watch API.

/cc @jakelandis

Relates #36008 #36035 #35849

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

We discussed this our area sync and came to the following conclusions:

• For 7.0.0 - Ensure the fully BWC change is implemented by default via the rest_total_hits_as_int as described on Make hits.total an object in the search response #35849
• For 7.0.0 - Ensure that there is a setting that can change the default behavior, such that consumers can effectively remove rest_total_hits_as_int and opt-in to the performance benefits from Do not compute hit counts by default #33028
• For 8.0.0 - Flip the default behavior such that rest_total_hits_as_int is not sent, but still allowing consumers to opt-in into 'legacy' behavior.
• Between 7.0.0 and 8.0.0 define a migration plan. Automation is ideal (i.e. re-write watches), but is tricky to get correct for all cases and we can't account for clients that store watches in source control that don't have a bi-directional sync. Depredations, warnings, blogs, etc. can all be used to help inform that this change is coming and to be prepared.

Will leave this issue open until:

• 7.0.0 concerns are resolved
• 8.0.0 blocker issue logged
• 7.0.0 -> 8.0.0 migration issue logged

[spinscale]

The first part is already implemented, see here for the search input and here

The second part is partially implemented, see here, when a search request is parsed. We just need to add a proper test for this, but it seems to me that this is already working and the user can set rest_total_hits_as_int to false in the search request.

[jakelandis]

It looks like the default for rest_total_hits_as_int is set to false in master. We will need to change the default to true for 7.x.

[jakelandis]

rest_total_hits_as_int has been defaulted to true in master (7.x) via #36697

[jakelandis]

closing this issue as resolved with

7.x passive default: #36697
8.0.0 blocker issue: #38387