[Fleet]: Does not allow package upgrade to disable TSDB

[lalit-satapathy]

Version: 8.8.0-snapshot and mode of run: elastic-package.

As part of the 8.8.0 TSDB rollout testing; need a mechanism to disable TSDB on a package.

Here is the usage model:

• Certain packages in 8.8.0 will have index_mode "time_series" enabled.
• If any issues found during the run, we should have an ability to disable TSDB on the package.

Two options to disable TSDB:

• Manual command to be run in dev tools (Fleet to confirm the command usage for this)
  • Need the confirmation that the manual commands provide a mechanism to go back from TSDB mode to non TSDB mode including rollover.
• Upgrade option to install a new package with TSDB disabled.

We find that the second option (Upgrade a new package with TSDB enabled) has some errors, which completely blocks this path for users and testing.

• Issue found in upgrading k8 package with TSDB disabled. https://github.com/elastic/obs-infraobs-team/issues/1007#issuecomment-1542320376
• Issue found in upgrading nginx package from TSDB enabled (1.12.0-beta) to TSDB disabled (1.12.1-beta)

We need this to be fixed to proceed with testing.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[lalit-satapathy]

CC: @ruflin @andresrc @mlunadia

[hop-dev]

this is the error we receive on the fleet side, digging into it now

[2023-05-11T11:12:08.087+01:00][ERROR][plugins.fleet] ResponseError: illegal_argument_exception
	Caused by:
		invalid_index_template_exception: index_template [metrics-nginx.stubstatus] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]
	Root causes:
		illegal_argument_exception: updating component template [metrics-nginx.stubstatus@package] results in invalid composable template [metrics-nginx.stubstatus] after templates are merged
    at KibanaTransport.request (/Users/markhopkin/dev/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    ```

[hop-dev]

An update:

The problem is that index_mode is being set on the index template itsself and the routing_path on the @package component template.
When we go to update the package we first update the component templates, then the index template afterwards. The problem with that is that for this particular update, we try to update the component template to remove routing_path but the validation correctly detects that index_mode is still set to time series, so rejects

I am looking at the implications of changing it so that we update the index template first, or move index mode to the component template

[kpollich]

move index mode to the component template

I think this should be our preference, but I recall having issues setting index_mode at the component template level during the initial implementation here. Perhaps that's not relevant anymore with stability improvements around TSDB features in Elasticsearch.

[ruflin]

A more general note: I thought we don't need to set routing_path anymore, so this might be a leftover?

[juliaElastic]

Fleet doesn't set routing_path, it is being generated by elasticsearch when the index.mode:time_series is set.

[juliaElastic]

I am looking at the implications of changing it so that we update the index template first, or move index mode to the component template

I think both of these are quite risky to change for a patch in 8.8.
We could add a fix to update the index template before the component template only if the time_series setting is being removed from an existing index template.

[juliaElastic]

The fix is ready and approved, should be auto merged when the build finishes.

[lalit-satapathy]

There is a new snapshot build. In the latest snapshot release, I dont get this error.

However, Index rollover did not happen after package upgrade. @kpollich index rollover should be automatically done right?

We are continuing our testing and need more time.

[juliaElastic]

It seems that the rollover is only happening when the TSDB is toggled on the integration policy: #149967

We can work on a fix, the question is, should this be flagged as a blocker for 8.8? As the last BC is being built today.

[lalit-satapathy]

We don't want to ask uses to do manual rollover, when a package upgrade is done. It is understandable as part of manual steps. Should not rollover work as part of the package upgrade?

[jlind23]

@juliaElastic Could you please work on a fix for this and raise it as a blocker in the 8.8 dev issue?

[juliaElastic]

As discussed with Julien, I think we wouldn't consider this as a blocker, since there is a workaround to manually roll over, and AFAIK this issue would happen for a few beta packages that has TSDB on and now turning it off in a new version. Any concerns @lalit-satapathy?

[ruflin]

This is a blocker for us to widely rollout TSDB. The basic assumption is, if there are packages that encounter issues with TSDB, we can quickly rollout a new version of the package with TSDB disabled. This means, users only need to upgrade a package. If this is not working, it means we need to ask all users that have this integration installed to manually rollover the data streams that are effected.

[andresrc]

@juliaElastic We are using Beta packages for testing, as soon as 8.8.0 is out we will start shipping GA packages, and some of them contain multiple data streams, we cannot tell users to manually roll over everything.

[juliaElastic]

Fix WIP: #157869