composable templates plan

[graphaelli]

Elasticsearch is introducing v2 of index templates per elastic/elasticsearch#53101. Beats don't rely on template inheritance (see elastic/beats#17829) so will likely not take into account APM Server's need for it.

elastic/beats#17809 is probably the best issue to coordinate with beats on APM Server needs. That issue appears to be headed for:

version	deployment	prefer_v2_templates
7.x	standalone	default (false)
7.x	agent	true
8.x	all	default (true)

This issue should be used for understanding the impact of v2 templates to APM Server and documenting the plans for support in the relevant version/deployment combinations. Let's open follow on issues for tracking the changes needed to execute on that plan.

[jalvz]

relevant: elastic/elasticsearch#56528

[simitt]

Related: having multiple dynamic templates with the same key will be disallowed in templates v2 and stop working. Related issues:

• open bug report in APM Server Using same key in dynamic template leads to overwrites #2693 (comment)
• beats issue for changing current template generation.
• ES#53326

[simitt]

see #3824 (comment)

[axw]

When we move to composable templates, I think it might be a good idea to have APM Server "own" the component templates it creates, and update them on upgrade - at least by default, not sure if it should be configurable. That way when APM Server is upgraded, users can automatically get new field mappings.

WDYT?

Never mind. Not particularly relevant, as we create separate templates for each server version.

[simitt]

After some more investigation I suggest to move forward as follows:

1. Remove fallback index in managed mode.
   When removing the fallback index we do not need an apm* template, which makes it easier to work with legacy and index templates. For legacy templates it avoids template inheritance. For index templates, if multiple templates matched we would need to work with different priorities.
   
   POC - removing fallback index for managed mode

2. ES 8.0 does not support template inheritance. Last minor version of APM Server 7.x needs to support Elasticsearch 8.0. We can refactor the current template loading to remove template inheritance of alternatively not make any changes for the legacy templates but require users to use the new composite and index template handling in last 7.x version when working with ES 8.0.
   
   Option a: Remove template inheritance and ensure only one index pattern matches an index.
   For unmanaged mode the goal is to only have one template with the default index pattern of apm* matching all apm indices. For managed mode we should have one template for the sourcemap and one for the onboarding index and only one template per event type. The default template and the ILM specific settings would be merged into one template per event type. When getting rid of the fallback apm index described in step (1), no apm* template is required.
   
   Users might switch between managed and unmanaged indices, which is supported out of the box right now (for default index settings). To keep this working existing templates can be deleted or the the index pattern is changed to something non-matching. Changing the index-pattern seems more friendly if someone needs to troubleshoot issues, as it keeps the templates and only changes the pattern. A suffix can be added to the index pattern, e.g. disabled-%{randomID}. For unmanaged mode, the disabling suffix needs to be added to the sourcemap, onboarding and all event type templates. For managed mode it would be added to the default apm template.
   
   Note: If index_suffix (managed) or indices (managed) were configured, switching between managed and unmanaged might require manual interaction, as the described changes might not be fully doable automatically (this is already the case today).
   
   Breaking change: In current versions the settings apm-server.ilm.setup.enabled and apm-server.ilm.setup.overwrite define whether the event specific templates only containing settings should be loaded. The setup.template.enabled and setup.template.overwrite settings define whether the default apm template including the mappings is loaded.
   When combinig everything into one template we cannot make this fine granular distinction. The template would most probably be laoded/overwritten if either one of the settings is true
   
   POC - removing template inheritance, changing index patterns to non-matching (note that the index-pattern is not random in the POC, but that would be a minor change to make).
   
   Option b: Require users to switch to index templates in last 7.x version when using it with an ES 8.0; possibly default to index templates in an earlier version. IMO this is perfectly fine.
   
   I suggest to move forward with Option b rather than introducing more changes and complexity by removing template inheritance and changing the semantics of currently used config options.

3. Implement new index templates and composable templates
   Create one composable template for apm wide mappings and settings and one composable template per event type with event specific settings. Then depending on indices being managed or unmanaged, create index templates using these composable templates.
   
   For unmanaged mode only one apm index template matching all apm indices should exist. Set the composed_of to only contain the template with generic apm mappings and settings.
   For managed mode one index template per event type is created, specifying the generic apm component and the event specific component template in the composed_of section. For sourcemap and onboarding indices one index template exists for each of them, referring only to the generic apm component template. When getting rid of the fallback index as suggested in step 1, no additional apm index template is needed.
   
   When switching between managed and unmanaged indices we can delete existing index templates, change the index pattern to something non-matching, or change the priority in the index template to that multiple templates that match the same index do not have the same priority, and the template that should match has the highest priority. I suggest to not delete the templates, but change the index pattern or the priority both is fairly easy achievable.
   
   Note: If index_suffix (managed) or indices (managed) were configured, switching between managed and unmanaged might require manual interaction, as the described changes might not be fully doable automatically (this is already the case today).
   
   Again, the existing settings proof to be tricky: combining setup.template.enabled and apm-server.ilm.setup.enabled
   as well as setup.template.overwrite and apm-server.ilm.setup.overwrite might lead to some user confusion. My suggestion is that the setup.template.* settings define what should be loaded for the general apm component template, while apm-server.ilm.setup.* defines wheter ILM specific component templates are loaded. For the index template either one of the settings need to be enabled to be loaded. (This needs some fine tuned consideration during the implementation phase).
   
   Switching from legacy templates to index templates seems to work without issues. Switching the other direction requires manual interaction and the index templates matching against the same indices as the legacy templates need to be removed. (Since the legacy template option is only kept for backwards compatibility this seems ok).
   
   POC - supporting legacy, component and index templates using the option to changing the index pattern to non-matching
   
   This also requires changes in libbeat: POC - adding component and index templates.

4. Make use of data streams when available: Set data_stream instead of rollover_alias when supported by ES version and license.
   ES data streams are intended for time series data only and for append-only use cases. Both prerequisites are ok for the APM use case.
   
   POC with datastreams
   
   Switching from ILM rollover alias to data stream does not raise an error, but if a rollover alias already exists with the same name, no data stream is created, but the rollover alias is used. This can lead to problems if the rollover alias is removed from the template when the data stream information is added.

5. Allow adding custom composable templates to index templates via APM Server config, e.g. add a composed_of option in the apm-server.ilm.setup.* or the setup.template.* option. The concrete config option depends on wether config options will be changed for 8.0.

6. Default to new index templates, mark legacy templates as deprecated

7. Decide on config options for 8.0:
   +apm-server.ilm.*
   +output.elasticsearch.index, output.elasticsearch.indices
   +setup.template.*

[axw]

For unmanaged mode only one apm index template matching all apm indices should exist. Set the composed_of to only contain the template with generic apm mappings and settings.

Couple of questions:

• should it also be composed of all the event-specific component templates? Otherwise it'll be missing field mappings...?
• have we considered moving away from having a single template for unmanaged mode?

It feels to me like there's a lot of complexity in retaining the existing configuration, particularly around switching between managed/unmanaged.

I'm wondering if we could simplify things, and make "unmanaged" truly unmanaged: do nothing. Otherwise always create index templates, with datastreams and ILM if available. Following this line of thought, could we deprecate both setup.template.* and apm-server.ilm.* altogether, and move the relevant bits under some new config?

To summarise, here's what I'm thinking:

1. Deprecate setup.template.* and apm-server.ilm.*
2. If any of setup.template.* or apm-server.ilm.* are explicitly specified, then we revert all behaviour to what we have today (until 8.0)
3. If ES version < 7.9.0, behaviour also remains the same as today
4. Introduce new config which controls index template management. If we didn't get caught by the above points, then:

• By default, create event specific index templates. Disable with something like index_templates.enabled: false
• Datastreams and ILM are always enabled if supported by the ES license level. If for whatever reason you don't want these enabled, then set index_templates.enabled: false and go your own way
• ILM policies can be defined under this new config, e.g. index_templates.ilm_policies: {...}
• Regardless of the value of index_templates.enabled, excluding legacy behaviour defined above, default to sending events to apm-%{[observer.version]}-%{[event.type]}-{index_suffix}. Users can add daily suffixes if they want, but we would strongly discourage (maybe even error out) if index_templates.enabled: true

One last point: I don't think switching between legacy/index templates would be supported -- make it a clean break. Instead, you update apm-server.yml before upgrading. If you accidentally upgrade and create legacy templates, then just delete them and restart APM Server after reconfiguring.