[EPM] Conditionally generate ES index pattern name based on dataset_is_prefix #89870

skh · 2021-02-01T14:49:31Z

Summary

Implements #88307

This now uses the following index patterns in the generated index templates. A manifest.yml file for a data stream status looking like this:

title: Apache status metrics
release: experimental
type: metrics
[...]

results in metrics-apache.status-*, and this:

title: Apache status metrics
release: experimental
type: metrics
dataset_is_prefix: true
[...]

results in metrics-apache.status.*-*

How to test this

Have a local registry running so that you can edit a data stream manifest.yml for testing.

Pick a package that contains a data stream without dataset_is_prefix set -- that would be any package we have in the registry now, I'm using the Apache package and the status data stream. Install it, and verify that the template for the data stream has been generated correctly:

in Kibana dev tools, look at the template list with GET _cat/templates and fetch a specific template for a data stream, e.g. with GET _index_template/metrics-apache.status
verify that the index_patterns property for this template looks like this: ["metrics-apache.status-*"] and that the priority of the template is 200.

Now uninstall the package and change the data stream manifest.yml to add the dataset_is_prefix: true setting. (You might need to restart the local registry after that change.)

in Kibana dev tools, fetch the template again
verify that the index_patterns property now looks like this: ["metrics-apache.status.*-*"] and that the priority of the template is 150.

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios

jalvz · 2021-02-02T09:51:59Z

APM fix for the issue described here: elastic/apm-server#4669

spalger · 2021-02-03T18:26:24Z

jenkins test this

skh · 2021-02-04T16:13:27Z

@elasticmachine merge upstream

elasticmachine · 2021-02-04T16:13:46Z

Pinging @elastic/fleet (Feature:EPM)

skh · 2021-02-04T16:36:16Z

Apologies for requesting so many reviews manually, I couldn't request one from @elastic/fleet.

jen-huang · 2021-02-04T20:02:54Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts

@@ -23,27 +23,29 @@ expect.addSnapshotSerializer({
 });

 test('get template', () => {


it would be great to have a unit test for installTemplate since that's the method that generates the index pattern and priority based on data stream information. I don't see any tests here that actually test on the dataset_is_prefix property

Good point. Is there an established way how to mock calls to elasticsearch in unit tests?

I've added unit tests for generateTemplateIndexPattern() and getTemplatePriority() as a start in 8b58f46

it looks like Kibana core does export some ES mocks, here's an example: https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/packages/_install_package.test.ts#L38

I see that installTemplate passes callCluster to other methods, so I guess you might have to mock ES return values in order of whenever callCluster is called further down the chain. the mocked ES client returns a jest.fn() so mocking the responses could be done like this:

const mockCallCluster = elasticsearchServiceMock.createLegacyScopedClusterClient().callAsCurrentUser .mockReturnValueOnce({ success: true }) .mockReturnValueOnce({ success: true }) .mockReturnValueOnce(// change mocked response whenever the response matters) ...

Thanks for the pointer, I've added a unit test for installTemplate().

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts

skh · 2021-02-08T11:39:59Z

@jen-huang Thanks for your review, this is ready for another look!

skh · 2021-02-08T12:57:09Z

@elasticmachine merge upstream

kibanamachine · 2021-02-08T15:03:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c7887ba

Metrics [docs]

✅ unchanged

History

💔 Build #104644 failed 0367454
💛 Build #104620 was flaky 4e543cb
💚 Build #104209 succeeded 31a83f41220e8cb473a68c3a13d360e9c6377cdb
💔 Build #104196 failed 8b58f46130659db90951620311752e23d3057ce3
💚 Build #103862 succeeded e678cfcadbd0f00070a73a05b205a372711fb511

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jen-huang

Thanks for adding those install tests. One small question but not a blocker, otherwise LGTM 👍🏻

jen-huang · 2021-02-08T15:30:26Z

x-pack/test/fleet_api_integration/apis/epm/template.ts

      });

      // This test is not an API integration test with Kibana
      // We want to test here if the template is valid and for this we need a running ES instance.
      // If the ES instance takes the template, we assume it is a valid template.
-      const { body: response1 } = await es.indices.putTemplate({
-        name: templateName,
+      const { body: response1 } = await es.transport.request({


why are the two ES requests here changed to the generic transport.request?

We're working with index templates v2. The endpoint for these is _index_template instead of _template, and the new ES client provides putIndexTemplate() and getIndexTemplate() methods to interface with that. The integration tests use the legacy client, which doesn't have these methods, so I fall back to the generic method.

At some point someone(TM) should probably tackle getting the new ES client into the integration tests, I didn't because it didn't look obvious at first glance.

For index templates v2 see elastic/elasticsearch#53101

(These tests were disabled for this reason, this PR reenables them.)

got it, thanks for that context. #74111 is the issue for migrating to new ES client, I just added a note about looking at usages in tests too

…s_prefix (elastic#89870) * Explicitly generate ES index pattern name. * Adjust tests. * Adjust and reenable tests. * Set template priority based on dataset_is_prefix * Refactor indexPatternName -> templateIndexPattern * Add unit tests. * Use more realistic index pattern in test. * Fix unit test. * Add unit test for installTemplate(). Co-authored-by: Kibana Machine <[email protected]>

…s_prefix (#89870) (#90656) * Explicitly generate ES index pattern name. * Adjust tests. * Adjust and reenable tests. * Set template priority based on dataset_is_prefix * Refactor indexPatternName -> templateIndexPattern * Add unit tests. * Use more realistic index pattern in test. * Fix unit test. * Add unit test for installTemplate(). Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

skh mentioned this pull request Feb 1, 2021

[Fleet] Alternative Index pattern in ES templates #88307

Closed

skh added Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.12.0 v8.0.0 labels Feb 1, 2021

skh force-pushed the 88307-use-dataset-is-prefix branch from 9778948 to ef78ab6 Compare February 2, 2021 16:10

jalvz mentioned this pull request Feb 2, 2021

Add service name to dataset field elastic/apm-server#4674

Merged

1 task

skh force-pushed the 88307-use-dataset-is-prefix branch from ef78ab6 to 1138547 Compare February 3, 2021 11:45

skh marked this pull request as ready for review February 4, 2021 16:13

skh self-assigned this Feb 4, 2021

skh requested review from ruflin, a team, nchaulet, jen-huang and jfsiii February 4, 2021 16:17

skh mentioned this pull request Feb 4, 2021

[EPM] Reassess default index template priority #90299

Closed

jen-huang reviewed Feb 4, 2021

View reviewed changes

skh force-pushed the 88307-use-dataset-is-prefix branch from e678cfc to 8b58f46 Compare February 5, 2021 11:13

ruflin reviewed Feb 5, 2021

View reviewed changes

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.test.ts Outdated Show resolved Hide resolved

skh added 5 commits February 8, 2021 11:20

Explicitly generate ES index pattern name.

9fb8dfa

Adjust tests.

0d45453

Adjust and reenable tests.

f5b4dcc

Set template priority based on dataset_is_prefix

0642673

Refactor indexPatternName -> templateIndexPattern

764b414

skh added 3 commits February 8, 2021 11:20

Add unit tests.

47f7556

Use more realistic index pattern in test.

890e2de

Fix unit test.

4e543cb

skh force-pushed the 88307-use-dataset-is-prefix branch from 31a83f4 to 4e543cb Compare February 8, 2021 10:20

Add unit test for installTemplate().

0367454

Merge branch 'master' into 88307-use-dataset-is-prefix

c7887ba

jen-huang self-requested a review February 8, 2021 15:27

jen-huang approved these changes Feb 8, 2021

View reviewed changes

skh merged commit c306a44 into elastic:master Feb 8, 2021

skh mentioned this pull request Feb 8, 2021

[7.x] [EPM] Conditionally generate ES index pattern name based on dataset_is_prefix (#89870) #90656

Merged

jalvz mentioned this pull request Feb 10, 2021

[Fleet] APM Server managed by Elastic Agent with Fleet - 7.12 elastic/apm-server#4558

Closed

15 tasks

skh mentioned this pull request Feb 15, 2021

Change recommended template priority in documentation elastic/elasticsearch#68982

Closed

jen-huang mentioned this pull request Feb 19, 2021

[Fleet] Set all keyword and text fields for index.query.default_field index template setting #91791

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EPM] Conditionally generate ES index pattern name based on dataset_is_prefix #89870

[EPM] Conditionally generate ES index pattern name based on dataset_is_prefix #89870

skh commented Feb 1, 2021 •

edited

Loading

jalvz commented Feb 2, 2021

spalger commented Feb 3, 2021

skh commented Feb 4, 2021

elasticmachine commented Feb 4, 2021

skh commented Feb 4, 2021

jen-huang Feb 4, 2021

skh Feb 5, 2021

jen-huang Feb 5, 2021

skh Feb 8, 2021

skh commented Feb 8, 2021

skh commented Feb 8, 2021

kibanamachine commented Feb 8, 2021

jen-huang left a comment

jen-huang Feb 8, 2021

skh Feb 8, 2021 •

edited

Loading

jen-huang Feb 8, 2021

		@@ -23,27 +23,29 @@ expect.addSnapshotSerializer({
		});

		test('get template', () => {

[EPM] Conditionally generate ES index pattern name based on dataset_is_prefix #89870

[EPM] Conditionally generate ES index pattern name based on dataset_is_prefix #89870

Conversation

skh commented Feb 1, 2021 • edited Loading

Summary

How to test this

Checklist

jalvz commented Feb 2, 2021

spalger commented Feb 3, 2021

skh commented Feb 4, 2021

elasticmachine commented Feb 4, 2021

skh commented Feb 4, 2021

jen-huang Feb 4, 2021

Choose a reason for hiding this comment

skh Feb 5, 2021

Choose a reason for hiding this comment

jen-huang Feb 5, 2021

Choose a reason for hiding this comment

skh Feb 8, 2021

Choose a reason for hiding this comment

skh commented Feb 8, 2021

skh commented Feb 8, 2021

kibanamachine commented Feb 8, 2021

💚 Build Succeeded

Metrics [docs]

History

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Feb 8, 2021

Choose a reason for hiding this comment

skh Feb 8, 2021 • edited Loading

Choose a reason for hiding this comment

jen-huang Feb 8, 2021

Choose a reason for hiding this comment

skh commented Feb 1, 2021 •

edited

Loading

skh Feb 8, 2021 •

edited

Loading