host.ip and source.ip have conflicting data types in the metrics-* index pattern

[hendry-lim]

Kibana version: 7.9.0-SNAPSHOT

Elasticsearch version: 7.9.0-SNAPSHOT

Original install method (e.g. download page, yum, from source, etc.): Docker

Describe the bug:
There are 2 conflicting fields, namely host.ip and source.ip in the default metrics-* index pattern.

Steps to reproduce:

1. Enroll and run elastic-agent-7.9.0-SNAPSHOT (with system integration enabled)
2. In Kibana, go to Stack Management --> Index Patterns
3. Open the metrics-* index pattern
4. Refresh the field list

Expected behavior:
host.ip and source.ip should be mapped to ip data type.

Screenshots (if relevant):

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[hendry-lim]

source.ip is still a conflict in 7.12.0.

[sgrodzicki]

Pinging @elastic/fleet

[skh]

This might be an issue with the integrations, cc @elastic/integrations

[ruflin]

The right type should be ip. @fearful-symmetry it seems linux package does not use it?

@jen-huang @mattkime One other issue with index patterns. In an ideal world, the above conflicts would not happen but in reality we will see this more and more often. The problem above comes from the fact that we have metrics-* and all packages install their fields there. In querying Elasticsearch this does not matter as it is much more granular. And if queried globally, AFAIK Elasticsearch supports the above case.

[fearful-symmetry]

A while ago I tried to purge all the errant keyword ip mappings from things, looks like I missed one. Should be an easy fix, at least.

[ruflin]

@fearful-symmetry Can you follow up on this with a fix?
@ycombinator @mtojek It would be nice if we could detect that a package uses an ECS mapping but has not defined it. Or going further, uses fields that are not defined and then if the field is defined and conflicts with ECS, the check fails.

[mattkime]

@ruflin Is there a something that you'd like to see changed in index patterns?

[fearful-symmetry]

Yep. Wanted to fix this last week, ended up getting sidetracked.

[fearful-symmetry]

Fixing this now, but the more that I look at this, the more I wonder what source.ip is doing in linux/users, since that's just...users. I assume it's a metadata processor or something somewhere?

[fearful-symmetry]

@hendry-lim ^

[fearful-symmetry]

Ah, never mind, it reports the IP if there's a remote user.

[ruflin]

@mattkime I think the problem lies here not necessarily directly with index pattern as it is correct that there is a conflict. It is the more fundamental issue that we require a metrics-* index pattern in the first place. The short term problem is that it shows a big error even for user ingested data but actually from a query perspective, all is fine. So maybe done done the error a bit or make it a config option on the index pattern if it should be shown or not?

[ycombinator]

@ycombinator @mtojek It would be nice if we could detect that a package uses an ECS mapping but has not defined it. Or going further, uses fields that are not defined and then if the field is defined and conflicts with ECS, the check fails.

This can be something handled by elastic-package lint. But for this the tool will first need to know which fields are ECS fields, i.e. a single source of truth for all ECS mappings / field definitions.

In fact, once we have such a single source of truth, we can take this a step further. Packages can then simply reference an ECS field name without needing to specify any more information about the field like its data type, description, etc. At build time (elastic-package build) the relevant mappings would get pulled from the single source of truth into the package's field definition files (e.g. automatically construct a fields/ecs.yml for that package's data streams). This would prevent conflicts such as the one in this issue from occurring in the first place.

WDYT @mtojek @ruflin? We can flesh this out further in a dedicated issue in the elastic-package repo if you agree with the general idea.

[ebeahan]

In fact, once we have such a single source of truth, we can take this a step further. Packages can then simply reference an ECS field name without needing to specify any more information about the field like its data type, description, etc. At build time (elastic-package build) the relevant mappings would get pulled from the single source of truth into the package's field definition files (e.g. automatically construct a fields/ecs.yml for that package's data streams). This would prevent conflicts such as the one in this issue from occurring in the first place.

@ycombinator This sounds fantastic! I think it would help alleviate some of the ECS field duplication burden mentioned in elastic/package-spec#63

[ycombinator]

Ah, thanks for linking to elastic/package-spec#63, @ebeahan. Sounds like that issue is proposing pretty much exactly what I proposed above as well!

[ruflin]

Lets continue the discussion in elastic/package-spec#63 around the validation. Thanks @ebeahan for referencing in.