Move non-ECS fields from root level event

[kfirpeled]

Motivation

Currently, having custom (non ECS) fields at the root level of vulnerability is a wrong doing
It can cause conflicts with other integrations

The proposed way for integrations having customized fields is to add them under the package name
In our case, cloud_security_posture.*

Definition of done

• Move/remove root level fields like: type, class, cluster_id and make sure cloudbeat's events follow the specified guideline
• Fields that already exist prior 8.8, for backward compatibility, add to the ingest pipeline rules to remove these fields / move them to the proper place (only if they are not being used in kibana)
• Create follow-up for custom fields we want to be part of ECS
• Explore if there are more fields that weren't mentioned in DOD

Out of scope

Related tasks/epics

[Omolola-Akinleye]

@kfirpeled is additional work needed in Kibana,? I remember the bug fix for calculating the unique transform key. vulnerability.package.name, vulnerability.package.version, vulnerability.package.fix_version are used in Kibana cc: @opauloh

[uri-weisman]

@kfirpeled is additional work needed in Kibana,? I remember the bug fix for calculating the unique transform key. vulnerability.package.name, vulnerability.package.version, vulnerability.package.fix_version are used in Kibana cc: @opauloh

Yeah, I noticed that those fields are omitted from the findings table.
Another option is just adding the package object to the root level and keeping the old fields as is for BC.

[kfirpeled]

@uri-weisman currently this task is marked as 8.8 however the PR #912 is labeled with backport skip.

If it is not going to be backported lets update the issue version candidate. And @Omolola-Akinleye the transform will be updated to use the new fields.
If it is going to be backported, we will fix the usages in kibana

[uri-weisman]

We decided not to remove the fields that are used in the UI just add new ones.
If we'll be able to point our Kibana code to the new fields it will be great but as a security measure, we'll just add new ones and not remove them.

[kfirpeled]

@uri-weisman thanks for the update
I suggest duplicating the data for now, having it both in package.* and in vulnerability.package.*
So don't revert it completely. This will save us time doing the transition in kibana for 8.9
Makes sense?

UPDATE:
We do use cluster_id in kibana in several places and haven't used orchestrator.cluster.id, so if we plan to move cluster_id as well. let me know

UPDATE 2:
I have a PR trying to address future compatibility of cluster_id removal in kibana: elastic/kibana#156458

[kfirpeled]

Hi @uri-weisman do we have followup tasks to complete the DOD in 8.9?

[orestisfl]

Verified

[uri-weisman]

Hi @uri-weisman do we have followup tasks to complete the DOD in 8.9?

@kfirpeled - no, do you want me to open the Kibana task of pointing to the new fields while supporting the previous version schema?

[kfirpeled]

Yes, thanks @uri-weisman
Can you say in which cloudbeat version orchestrator.cluster.id was added?

And what is the scope of the change? instead of vulnerability.package to use package.*? and instead of cluster_id to use orchestrator.cluster.id?

Anything else?

[uri-weisman]

Can you say in which cloudbeat version orchestrator.cluster.id was added?

8.8.0

• [Cloud Security] Use new fields locations in CNVM findings kibana#157674