Support legacy template api in elasticsearch 8

[kaisecheng]

Fixed: #1088

Prior to the change, the plugin uses legacy API for ES 7.x and forces to use index template API for ES 8.x. However, legacy template API is still available in ES 8 and user should be allowed to use it.

This commit adds a new flag template_api to control which API to use. The available value is auto, legacy and composable. The default value is auto.

• auto Logstash detect the version of Elasticsearch. It uses index template API for 8 and legacy API for 7.
• legacy use legacy API
• composable use index template API

[yaauie]

Implementation looks clean and sensible.

I'd like to take a small step back and discuss the interface that we are exposing to the user

Right now the interface of this centered on the old template api:

• template_legacy => true: use the old thing
• template_legacy => false: don't use the old thing
• template_legacy => auto: use the old thing sometimes?

If we are stuck on a boolean-with-auto, I would prefer to either invert the setting to be centered on the new API, so that a user can tell us:

• template_modern => true to use the new thing
• template_modern => false to not use the new thing
• template_modern => auto to assume which one to use

But even this introduces terminology like "modern" not found in Elasticsearch documentation, and I particularly don't like "boolean" options with 3+ choices.

Elasticsearch mainly describes V2 templates as just plain "Index templates" (often in direct opposition to "legacy templates"), but it also uses "composable" to refer to the v2 templates when describing how they interact with the legacy templates:

Composable templates take precedence over legacy templates. If no composable template matches a given index, a legacy template may still match and be applied.

Maybe template_format with options legacy, composable, and auto?

• template_format => legacy: use the old thing
• template_format => composable: use the new thing
• template_format => auto: idk just do your best?

Or similarly template_api?

• template_api => legacy: use the old thing
• template_api => composable: use the new thing
• template_api => auto: idk just do your best?

---

Separately, when the effective value is auto, and we are talking to an ES7, I think we should emit a deprecation notice informing the user that they should either "lock in" legacy templates in advance of upgrading any part of their stack, or that they should migrate to composable templates.

[mashhurs]

Implementation looks clean and sensible.

I'd like to take a small step back and discuss the interface that we are exposing to the user

Right now the interface of this centered on the old template api:

• template_legacy => true: use the old thing
• template_legacy => false: don't use the old thing
• template_legacy => auto: use the old thing sometimes?

If we are stuck on a boolean-with-auto, I would prefer to either invert the setting to be centered on the new API, so that a user can tell us:

• template_modern => true to use the new thing
• template_modern => false to not use the new thing
• template_modern => auto to assume which one to use

But even this introduces terminology like "modern" not found in Elasticsearch documentation, and I particularly don't like "boolean" options with 3+ choices.

Elasticsearch mainly describes V2 templates as just plain "Index templates" (often in direct opposition to "legacy templates"), but it also uses "composable" to refer to the v2 templates when describing how they interact with the legacy templates:

Composable templates take precedence over legacy templates. If no composable template matches a given index, a legacy template may still match and be applied.

Maybe template_format with options legacy, composable, and auto?

• template_format => legacy: use the old thing
• template_format => composable: use the new thing
• template_format => auto: idk just do your best?

Or similarly template_api?

• template_api => legacy: use the old thing
• template_api => composable: use the new thing
• template_api => auto: idk just do your best?

Separately, when the effective value is auto, and we are talking to an ES7, I think we should emit a deprecation notice informing the user that they should either "lock in" legacy templates in advance of upgrading any part of their stack, or that they should migrate to composable templates.

template_api looks good for me. And one thing I would like to propose is attaching index prefix to the naming (becomes index_template_api). The actual change of why ES deprecated _template is, new component templates are introduced. Let's make sure in the future we will have a window if something comes up for component templates.

[kaisecheng]

template_api with value legacy, composable and auto is my favorite. The keywords align with Elasticsearch, which is very important. Thanks for the suggestion @yaauie

@mashhurs The reason I avoid adding index prefix is to align the name with others template, template_name, template_overwirte and manage_template. They are all meant to manage the index template. Thanks for your input.

[yaauie]

Implementation looks great. I have a couple of suggestions on wording for both the log warning and the docs page.

[yaauie]

🐙LGTM