Add index.query.default_field setting to index templates in 6.7

[joshdover]

Related to elastic/kibana#32769 and elastic/elasticsearch#39851

tl;dr: The index.query.default_field index setting for index templates is gated for 7.0+. We should add this for 6.7 to support a smoother upgrade process.

Metricbeat indices created before Elasticsearch 7.0 are missing the index.query.default_field setting which breaks searching those indices without specifying a field to search on (non-qualified searches) in 7.0, a common usage pattern with Kibana's query bar. This setting is currently gated for 7.0 here.

Elasticsearch 6.7 will be adding a deprecation warning for any indices with >1024 fields that do not have index.query.default_field specified. In Kibana, we will show a button for any of these deprecation warnings that apply to Metricbeat indices that will allow the user to automatically add the setting with the default list of Metricbeat fields that are on new Metricbeat indices. This solves the problem for historical data, but not for newly created indices.

I believe this setting should be added to Beats 6.7 to make upgrades smoother. This index setting has existed in Elasticsearch since at least 5.6 so I'm not quite sure why it wasn't added to 6.x Beats originally.

Reasons to do this

From the Kibana side:

If we do not, then the user will have new indices being created that will need to be changed only after they upgrade to 7.0 which could create issues (in particular, a seemingly broken Kibana). Of course, they wouldn't be losing any data, but it seems like unnecessary support burden for an easily fixed bug on our end.

For context, we're trying to make it so that the user does not have to return to the Upgrade Assistant after upgrading. The only edge we have right now is for APM indices, which require a ECS migration after upgrading to 7.0. In that case, the user is presented with a banner from within the APM app that links to the Upgrade Assistant to resolve the issue for historical data. I'd like to avoid this for Beats entirely as there is no dedicated place in the UI for Beats data.

From @ruflin on the Beats side:

Having it in 6.7 will also solve an other edge case. Let's assume someone used module foo which populates field foo. Now the migration to 7.0 is happening and default_fields are applied. Now the users enabled module bar in 6.7 and also the field bar is populated. As I assume Kibana will apply the setting for fields which actual data, these new fields would be missing. If we have it in 6.7, we don't hit this issue. As everything before 6.7 can't ship data to ES 7.0, the older indices will not receive any new data.

---

I believe this should be blocker because it breaks a common experience: using the Kibana search bar on a metricbeat index. If a user has any 6.x metricbeat indices that match their index pattern, after upgrading to 7.0, using the search bar without specifying a field will blow up in Kibana.

[ph]

Looking briefly at the code we have the logic in place to setup the default_fields, we currently gate this option for 7.x. One thing, I do not see where defaultFields is set, it's currently a package variable but its not exported.

beats/libbeat/template/template.go

Lines 301 to 308 in 265ab8b

	if ver.Major >= 7 {
	// copy defaultFields, as defaultFields is shared global slice.
	fields := make([]string, len(defaultFields))
	copy(fields, defaultFields)
	fields = append(fields, "fields.*")
	indexSettings.Put("query.default_field", fields)
	}

This probably mean that we have other beats with the same problem or all of thems?

@ruflin I am leaving this with you, feel free to downgrade.

[joshdover]

@ph I was previous pointed to this code which appears to make any keyword, text, or ip field part of the default_field setting: https://github.com/elastic/beats/blob/master/libbeat/template/processor.go#L104

[ruflin]

I opened #11205. Please have a look at my comment there and if we all agree it's good to get this in, let's merge.

[ruflin]

Closing as #11205 was merged.