Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions #455

mrodm · 2022-12-16T12:27:43Z

What does this PR do?

This PR adds support in input packages to include under elasticsearch key dynamic templates and properties (mappings). This PR allows to define elasticsearch.index_template.* keys. It uses the same definition as the data_stream manifests in integration packages.

This PR also adds a new field under _dev/build/build.yml named import_common_dynamic_mappings (other options import_common_mappings?). The aim of this field is to allow developers to opt in to be added automatically by elastic-package
some common definitions needed.

This support would be introduced in package-spec versions >= 2.3.0.

Additional notes:

Updated .gitignore to just ignore build folder at the root directory. There are several build folders used in our specs, and git raises some errors because of this:

 $ git add spec/integration/_dev/build/build.spec.yml
The following paths are ignored by one of your .gitignore files:
spec/integration/_dev/build
hint: Use -f if you really want to add them.
hint: Turn this message off by running
hint: "git config advice.addIgnoredFile false"

According to the Jenkins outputs it would be needed to just ignore the one from the root level:

[2022-11-23T10:15:49.793Z] mkdir -p /var/lib/jenkins/workspace/gest-manager_package-spec_PR-452/src/github.com/elastic/package-spec/build/test-coverage/coverage-unit-report
[2022-11-23T10:15:49.793Z] mkdir -p /var/lib/jenkins/workspace/gest-manager_package-spec_PR-452/src/github.com/elastic/package-spec/build/test-results

Why is it important?

This would allow to elastic-package to add the most common mapping definitions needed to avoid conflicts in mappings for both input and integrations (data_strreams) packages.

Checklist

I have added test packages to test/packages that prove my change is effective.
I have added an entry in spec/changelog.yml.

Related issues

Relates Temporarily import hard-coded list of common fields elastic-package#1018

…and properties from ECS Remove elasticsearch field in input manifest from <2.3.0 versions

elasticmachine · 2022-12-16T12:37:40Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-20T14:55:04.625+0000
Duration: 9 min 32 sec

Test stats 🧪

Test	Results
Failed	0
Passed	678
Skipped	0
Total	678

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-12-16T12:37:47Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`7/7`)	💚
Files	68.0% (`17/25`)	👍
Classes	76.471% (`26/34`)	👍
Methods	56.075% (`60/107`)	👍
Lines	42.229% (`538/1274`)	👍
Conditionals	100.0% (`0/0`)	💚

mrodm · 2022-12-20T10:28:24Z

code/go/pkg/validator/validator_test.go

@@ -28,6 +28,7 @@ func TestValidateFile(t *testing.T) {
 	}{
 		"good":                               {},
 		"good_v2":                            {},
+		"good_input":                         {},


Added a new good_input package to add these keys. It is copied from the sql_input test package.

mrodm · 2022-12-20T10:29:40Z

test/packages/good_v2/manifest.yml

@@ -1,4 +1,4 @@
-format_version: 2.0.0


Update version in good_v2 to allow adding the new field in _dev/build/build.yml

jsoriano · 2022-12-20T11:16:17Z

spec/integration/_dev/build/build.spec.yml

+    patch:
+      - op: remove
+        path: "/properties/dependencies/properties/ecs/properties/import_common_dynamic_mappings"


👍 this will encourage users of this to update the format of their packages 🙂

jsoriano · 2022-12-20T11:19:18Z

spec/integration/_dev/build/build.spec.yml

@@ -20,7 +20,17 @@ spec:
              type: string
              description: Source reference
              pattern: '^git@.+'
+            import_common_dynamic_mappings:


Nit. common looks redundant here.

Suggested change

import_common_dynamic_mappings:

import_dynamic_mappings:

And maybe dynamic too, this feature imports mappings for this schema, they happen to be mostly dynamic, but could be static too.

Suggested change

import_common_dynamic_mappings:

import_mappings:

Or:

Suggested change

import_common_dynamic_mappings:

import_template:

I was not totally sure about this name. Agreed that common does not provide any value to the name.
I will update this field to be import_mappings

Thanks!

spec/changelog.yml

hop-dev

Addition of the elasticsearch key at the top level of input packages looks good to me 👍 Thanks for this Mario!

mrodm added 4 commits December 16, 2022 12:40

Add definition for elasticsearch.index_template property

2adcc70

Add elasticsearch field into input manifest

c3b46bc

Add parameter to opt-in about adding or not common dynamic templates …

dc5b95a

…and properties from ECS Remove elasticsearch field in input manifest from <2.3.0 versions

Just ignore build folder at the root of the repo

5ae48af

mrodm self-assigned this Dec 16, 2022

Add changelog entry

2056f3c

mrodm added 5 commits December 16, 2022 14:11

Fix reference path

aedf843

Add json patch to remove import_common_dynamic_mappings before 2.3.0

291a635

Update test package good_v2

9a32db0

Add a reference good_input package

8c21116

Add good_input to be tested

0dcd837

mrodm mentioned this pull request Dec 20, 2022

Add common dynamic mappings and properties from ECS automatically elastic/elastic-package#1073

Merged

mrodm commented Dec 20, 2022

View reviewed changes

mrodm marked this pull request as ready for review December 20, 2022 10:30

mrodm requested a review from a team as a code owner December 20, 2022 10:30

mrodm requested a review from hop-dev December 20, 2022 10:30

jsoriano reviewed Dec 20, 2022

View reviewed changes

mrodm added 3 commits December 20, 2022 15:49

Add changelog entry for flag

a1bde88

Rename field for import mappings

70d4563

Update test packages

6f56801

mrodm requested a review from jsoriano December 20, 2022 18:03

jsoriano approved these changes Dec 21, 2022

View reviewed changes

mrodm mentioned this pull request Dec 22, 2022

Temporarily import hard-coded list of common fields elastic/elastic-package#1018

Closed

3 tasks

hop-dev approved these changes Dec 22, 2022

View reviewed changes

hop-dev mentioned this pull request Dec 22, 2022

[Fleet] Input only packages: Create index templates and pipelines at package policy creation time elastic/kibana#145529

Closed

mrodm merged commit 342bdf7 into elastic:main Dec 22, 2022

mrodm deleted the add_elsaticsearch_field_input branch December 22, 2022 13:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions #455

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions #455

mrodm commented Dec 16, 2022 •

edited

Loading

elasticmachine commented Dec 16, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 16, 2022

mrodm Dec 20, 2022

mrodm Dec 20, 2022

jsoriano Dec 20, 2022

jsoriano Dec 20, 2022

jsoriano Dec 20, 2022

mrodm Dec 20, 2022

hop-dev left a comment

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions #455

Add elasticsearch mappings definitions into input packages and a key to opt-in to import common definitions #455

Conversation

mrodm commented Dec 16, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Dec 16, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Dec 16, 2022

🌐 Coverage report

mrodm Dec 20, 2022

Choose a reason for hiding this comment

mrodm Dec 20, 2022

Choose a reason for hiding this comment

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

jsoriano Dec 20, 2022

Choose a reason for hiding this comment

mrodm Dec 20, 2022

Choose a reason for hiding this comment

hop-dev left a comment

Choose a reason for hiding this comment

mrodm commented Dec 16, 2022 •

edited

Loading

elasticmachine commented Dec 16, 2022 •

edited

Loading