[System] Update cpu, load, memory dataset with dimensions

[ruflin]

We are in the progress of starting to adopt TSDB. To enabled TSDB, dimension fields are required. In this PR, dimensions are added to the following dataset:

• system.cpu
• system.load
• system.memory

The dimensions that were selected are:

• agent.id
• container.id
• host.name
• cloud.instance.id
• cloud.project.id
• cloud.provider

These 4 dimensions should set a good foundation for the dataset selected above. It should make it work well in host, cloud and container environments. Not all dimensions always have to exist. We still need to define a more generic set of dimensions for integrations.

The goal of this PR is to enable us testing TSDB on common dataset. It also allows Kibana developers to have dataset ready for testing.

As long as TSDB is not enabled and on older versions of the stack, this change will not have any effect.

[ruflin]

This field also exists in fields.yml, seems it was a duplicate.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-02T07:45:31.843+0000

• Duration: 16 min 44 sec

Test stats 🧪

Test	Results
Failed	0
Passed	194
Skipped	0
Total	194

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (3/3)	💚
Files	100.0% (4/4)	💚
Classes	100.0% (4/4)	💚
Methods	60.759% (48/79)	👍 27.426
Lines	99.533% (2771/2784)	👎 -0.467
Conditionals	100.0% (0/0)	💚

[agithomas]

When we add a field as a dimension would there be performance benefits while querying the data ?

If yes, we may need to rethink what must go as a dimension fields.

using instance name instead of instance id, container name instead of container ID may be useful from a performance standpoint.

[martijnvg]

When we add a field as a dimension would there be performance benefits while querying the data ?

I don't think there is a direct query performance benefits from less or more dimensions. It is important that it uniquely defines a time serie. Internally we use dimensions to create a tsid and the tsid is used for routing and index sorting. So it does have a direct impact on how well the data gets stored (similar metrics that get routed to the same shard and sorted next together in that shard compress better, which means lower disk space usage, less memory usage, which then can positively impact query performance).

[agithomas]

Wouldn't it be safe to additionally include - cloud.project.id, cloud.provider as dimension fields ?

Not all apps will be container based. There could be monolithic apps that run on VMs. Host names can remain same in a multi-regional setup. In such scenario, including cloud.project_id may be needed.

[ruflin]

Good idea, I added the two additional fields.

[agithomas]

LGTM

[leehinman]

LGTM. nit on the version number.

One side question, many of these fields are ECS fields and we could/should be using the external definition so all of the descriptions are kept up to date automatically. Any idea how adding the demension key works with that?

[ruflin]

One side question, many of these fields are ECS fields and we could/should be using the external definition so all of the descriptions are kept up to date automatically. Any idea how adding the demension key works with that?

I had quickly tried that and it didn't work. Somehow my dimension setting was overwritten. But I think it is the path we need to investigate I just didn't want to block this change here on it.