Support v2 templates in ES indexing calls

[ycombinator]

Elasticsearch is introducing v2 of index templates. To understand the motivation for this change and it's details, see: elastic/elasticsearch#53101.

In 7.x (once elastic/elasticsearch#55411 is merged), Elasticsearch indexing APIs will accept an optional query string parameter: prefer_v2_templates. In 7.x, the default value for this parameter will be false, indicating that v1 templates should be used. In 8.0, the default value will change to true, indicating that v2 templates should be used.

This issue is to track changes needed in Beats to work with v2 templates. Concretely:

• For 7.x Beats:

  • If the Beat is operating standalone (i.e. not managed via Fleet + Agent), it is responsible for creating the index template (v1) in Elasticsearch. As such, indexing calls from such Beats should pass prefer_v2_templates=false in indexing requests to Elasticsearch.
  • If the Beat is being managed by Fleet, then the Beat is not responsible for creating the index template in Elasticsearch. It assumes (maybe checks?) the index template has been created before indexing it's events. Further, Fleet will create v2 index templates. So such Beats should pass prefer_v2_templates=true in indexing requests to Elasticsearch.

• Starting 8.0.0, Beats will use v2 index templates, either ones they create on their own (in standalone mode) or because Fleet created them beforehand. So such Beats should pass prefer_v2_templates=true in indexing requests to Elasticsearch.

Related: #17829

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[urso]

If the Beat is operating standalone (i.e. not managed via Fleet + Agent), it is responsible for creating the index template (v1) in Elasticsearch. As such, indexing calls from such Beats should pass prefer_v2_templates=false in indexing requests to Elasticsearch.

In case the generated templates are the same from Beats POV, I would opt to not send a prefer_v2_templates flag at all, no matter which ES version is used. Instead we should only use prefer_v2_templates=true, iff the Beat is managed by Fleet + Elasticsearch is < 8.0. WDYT?

[ycombinator]

Agreed on both points, assuming Beats does not need to do any v1->v2 migration work (which seems quite likely to be the case, but will know for sure after #17829) .

[ph]

@urso @ycombinator what would be the best way to enable that? Are you planning to have an option in the output that we can just set?

[ycombinator]

@ph Yes, that could be one way: an option in the elasticsearch output that Agent can set. We can leave this option undocumented if we don't want end-users to use it. Alternatively, we could add a new class of settings to Beats under a fleet.* namespace if we expect there to be more "internal API" settings that only make sense between Fleet and Beats.

[ph]

Good point, I think it depends at how we want to expose this to other use case. If we do not, we could add a new option under management.v2_templates: true or management.fleet.v2_template: true

This seem to be a temporary usage so I would go on the easiest way to implement it.

[ycombinator]

This seem to be a temporary usage so I would go on the easiest way to implement it.

Good point — this option will go away at some point, probably 9.0. Easiest would be to make it an option on the ES output (potentially undocumented). So I'm +1 for this unless @urso has a different preference.

[urso]

No real preference :)

We also have a 'private' CLI flag named -environment. We currently use it to decide on defaults in the log output.

In case we really introduce a management or fleet namespace for other things as well, then it might be as simple as enable template v2 support once the namespace is available.

[urso]

Looks like the 'API' to install templates is changing as well:

V1 and V2 have separate APIs:

V1:
GET/PUT/DELETE /_template/<name>

V2:
GET/PUT/DELETE /_component_template/<name>
GET/PUT/DELETE /_index_template/<name>

[ycombinator]

@urso Yes, I made the same comment on the related issue: #17829. This issue here is meant just for the indexing part.

[ph]

We have added a workaround in the 7.8/7.x released of the agent.
We uses the params option to the elasticsearch output for that.

[urso]

The indexing part did become much easier for us (prefer_v2_template parameter has been removed): elastic/elasticsearch#56528

[ph]

@urso indeed its much simplier now. thanks for closing this.