[Prototype] Test package with TSDB and synthetic source enabled

[ruflin]

This is a change to the system.memory dataset in the system package. The intention is to use time_series index, define the dimensions and turn synthetic source on. This follows the change in elastic/package-spec#419.

This PR is for testing and discussion purpose, it is not expected that this PR will ever be merged, this is why is is only in draft.

[ruflin]

I'm currently testing on the 8.7.0-SNAPSHOT from today. Based on elastic/kibana#144974 I assumed the TSDB part works. Looking at the UI it does:

But looking at the _index_template (_index_template/metrics-system.memory) and component template (_component_template/metrics-system.memory@package) the setting does not show up.

@juliaElastic Is the setting in some other template I missed? BTW same is true for synthetic source, but it is wrong in the UI and not in the templates.

Good news is, the time series dimensions show up as expected.

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-12-01T13:06:56.161+0000

• Duration: 11 min 9 sec

Steps errors

Expand to view the steps failures

Check integration: system

• Took 0 min 2 sec . View more details here
• Description: ../../build/elastic-package check -v

Google Storage Download

• Took 0 min 0 sec . View more details here

Google Storage Download

• Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ruflin]

@jsoriano Can I get your on this PR to ensure I use all the configs in the right way?

[ruflin]

According to elastic/kibana#141211 source_mode support is still in the works. @nchaulet As soon as you have the changes, lets test it with this package.

[juliaElastic]

But looking at the _index_template (_index_template/metrics-system.memory) and component template (_component_template/metrics-system.memory@package) the setting does not show up.

Is the setting in some other template I missed? BTW same is true for synthetic source, but it is wrong in the UI and not in the templates.

@ruflin You should see the generated settings in metrics-system.memory Index template settings, is it not working?

If not, can you check the HTTP request that goes out when submitting the integration policy, to verify that the TSDB flag is set correctly?

[ruflin]

I'm never hitting "save" on any policy. It is specified in the package itself: https://github.com/elastic/integrations/pull/4749/files#diff-8f3978582b1a5a9f6453c6025fc51811c87d64c3e82449575fe5ea3078fdc743R32 The change on the settings must happen during installation of the package.

[juliaElastic]

@ruflin so you are installing a package without an integration policy? I'm afraid this was not implemented, we only set the index.mode: time_series setting on saving an integration policy.
I didn't see this scenario listed in the implementation issue, so I wasn't aware elastic/kibana#144530

We can raise another issue to fix this. cc @kpollich
Btw synthetic_source works the same way, only changing component templates on package policy create/update.

[kpollich]

You are right, @juliaElastic - I only specified the following

If the current package's manifest contains index_mode: time_series for a given data stream, the toggle should be in an "enabled + readonly" state, e.g. it cannot be disabled

When in reality, I should've specified that the package manifest value should be honored at installation time as well, not just reflected in the UI.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ruflin]

Development on this will continue as soon as elastic/kibana#141211 lands.

[ruflin]

Closing this in favor of #5160 For inspiration on how to use TSDB, take a look at #4966