Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add reference to ecs@mappings for each index template #174905

Closed
Tracked by #8542
gsantoro opened this issue Jan 16, 2024 · 1 comment · Fixed by #174855
Closed
Tracked by #8542

Add reference to ecs@mappings for each index template #174905

gsantoro opened this issue Jan 16, 2024 · 1 comment · Fixed by #174855
Assignees
@gsantoro
Labels
Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@gsantoro
Copy link
Contributor

We want to add a reference to the component template ecs@mappings for any index template for any integration.

This component template is already installed by default by Elasticsearch, and it should be preferred to importing ECS fields in an integration.

This should simplify creating a new integration since you don't need to define or import as many fields.

It should be Fleet adding this extra reference in the code.

The same behaviour is valid for both logs and metrics index templates.

Linked to elastic/integrations#8542
@gsantoro gsantoro mentioned this issue Jan 16, 2024
Merged
2 tasks
@botelastic botelastic bot added the needs-team Issues missing a team label label Jan 16, 2024
@gsantoro gsantoro self-assigned this Jan 16, 2024
@gsantoro gsantoro mentioned this issue Jan 16, 2024
Closed
@dmlemeshko dmlemeshko added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Jan 16, 2024
@zmoog zmoog mentioned this issue Jan 31, 2024
Closed
gsantoro added a commit that referenced this issue Jan 31, 2024
@gsantoro @kpollich @felixbarny
Add reference to ecs@mappings for each index template (#174855)
8c5b508 
## Summary

I added a reference to the component template `ecs@mappings` when
building any index template for an integration.

The same behaviour is valid for both logs and metrics index templates.

Linked to elastic/integrations#8542

Close #174905

### Checklist

Delete any items that are not applicable to this PR.

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kyle Pollich <[email protected]>
Co-authored-by: Felix Barnsteiner <[email protected]>
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@gsantoro @kpollich
@felixbarny @CoenWarmer
Add reference to ecs@mappings for each index template (elastic#174855)
321a565 
## Summary

I added a reference to the component template `ecs@mappings` when
building any index template for an integration.

The same behaviour is valid for both logs and metrics index templates.

Linked to elastic/integrations#8542

Close elastic#174905

### Checklist

Delete any items that are not applicable to this PR.

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kyle Pollich <[email protected]>
Co-authored-by: Felix Barnsteiner <[email protected]>
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@gsantoro @kpollich
@felixbarny @fkanout
Add reference to ecs@mappings for each index template (elastic#174855)
5a92683 
## Summary

I added a reference to the component template `ecs@mappings` when
building any index template for an integration.

The same behaviour is valid for both logs and metrics index templates.

Linked to elastic/integrations#8542

Close elastic#174905

### Checklist

Delete any items that are not applicable to this PR.

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Kyle Pollich <[email protected]>
Co-authored-by: Felix Barnsteiner <[email protected]>
@zmoog zmoog mentioned this issue Mar 7, 2024
Merged
7 tasks
zmoog added a commit to elastic/elastic-package that referenced this issue Mar 18, 2024
@zmoog @jsoriano @mrodm
Validate fields using ECS JSON schema when all stack versions support…
30b16dc 
… ecs@mappings (#1711)

Enable `elastic-package` to import the ECS JSON schema to validate fields in two conditions:

- When package developers enabled import_mappings (existing behavior).
- When package developers did NOT enable import_mappings, but all stack versions in the Kibana constraints support the `ecs@mappings` in Elasticsearch (8.13.0+).

If the package supports one or more stack versions without ECS mappings, devs must support ECS using `import_mappings` (deprecated) or `ecs.yml` file. 

On 8.13, we [enabled](elastic/kibana#174905) Fleet to include the `ecs@mappings` component template in all the integration index templates.  So, starting from 8.13, integration developers can stop using `import_mappings` and the `ecs.yml`. 

However, `elastic-package` should import the ECS JSON schema only if ALL stack versions in the Kibana constraints have ECS mappings from Elasticsearch. 

* Continue support import mappings (deprecated only)

We should continue support import mappings for integrations that want
to.

We don't want to force integrations devs to migrate.

* Trying to infer if all stack versions support ECS

This is an experiment to try inferring if all supported stack versions
in the Kibana constraints support ECS mappings.

If not, we don't include the ECS JSON schema mappings. In this case,
users can opt for one of the following:

- enable import mappings
- add field mappings
- bump the Kibana constraints >= 8.13.0

* Use all ES releases < 8.13.0 to infer ECS support

We compare all ES releases without ECS support against the kibana
constraints to assess if the package can use the ECS JSON schema or
not.

There are no package versions for older stacks.

Sort the stack version numbers (`sort -V` is your friend here).

* Pre-parse all stack versions and move them out fn

* Reverse stack version order to improve efficiency

Since most of current integrations support 8.x:

$ cat all_kibana_version.txt | awk -F',' '{print $3}' | sort | uniq -c
   3 ^7.14.0 || ^8.0.0
   5 ^7.14.1 || ^8.0.0
   1 ^7.14.1 || ^8.8.0
  10 ^7.16.0 || ^8.0.0
   4 ^7.17.0 || ^8.0.0
   6 ^8.0.0
   7 ^8.10.1
   6 ^8.10.2
   3 ^8.11.0
   2 ^8.11.2
  75 ^8.12.0
   1 ^8.12.1
   1 ^8.12.2
  44 ^8.13.0
   1 ^8.2.0
   2 ^8.2.1
   6 ^8.3.0
   2 ^8.4.0
   4 ^8.5.0
   1 ^8.5.1
   2 ^8.6.0
   1 ^8.6.1
   1 ^8.7.0
  24 ^8.7.1
  27 ^8.8.0
   1 ^8.8.1
   1 ^8.8.2
  11 ^8.9.0

It's more efficient to check for recent stack versions.

* Update internal/fields/validate_test.go

Co-authored-by: Jaime Soriano Pastor <[email protected]>


* Add upcoming 7.17.x patch releases

Adding a few extra releases to support unlikely yet possible edge
cases.

* Add package to test ecs@mappings support


Co-authored-by: Mario Rodriguez Molins <[email protected]>

* Log on importing JSON schema from external source

---------

Co-authored-by: Jaime Soriano Pastor <[email protected]>
Co-authored-by: Mario Rodriguez Molins <[email protected]>
@andrewkroh andrewkroh mentioned this issue Jun 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gsantoro gsantoro

Labels
Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add reference to ecs@mappings for each index template gsantoro/kibana
3 participants
@gsantoro @dmlemeshko @elasticmachine