[Cloud Security] [CNVM] transform cuts off plenty of documents and views partial

[jeniawhite]

Describe the bug
Deploying the vulnerability solution with FE, we saw that the transform removed plenty of findings.
This might be due to uniqueness problems (need to be investigated).

[kfirpeled]

I found several issues that might lead the transform to fail

1. Uniqueness was done on a field that does not exist: unique_key: package.version instead of vulnerability.package.version
2. Uniqueness was done on a fields which lack mappings: vulnerability.package.version and resource.id

I'll create these two PRs and check if the issue was fixed or not

[kfirpeled]

After ~2.5h waiting for a single vulnerability when running cloudbeat locally I copied documents from the current demo environment to my local environment and everything seems to work fine - passing it to review

[jeniawhite]

Looking at the transformation, I saw the following:

export const latestVulnerabilitiesTransform: TransformPutTransformRequest = {
  transform_id: 'cloud_security_posture.vulnerabilities_latest-default-8.8.0',
  description:
    'Defines vulnerabilities transformation to view only the latest vulnerability per resource',
  source: {
    index: VULNERABILITIES_INDEX_PATTERN,
  },
  dest: {
    index: LATEST_VULNERABILITIES_INDEX_DEFAULT_NS,
  },
  frequency: '5m',
  sync: {
    time: {
      field: 'event.ingested',
      delay: '60s',
    },
  },
  retention_policy: {
    time: {
      field: '@timestamp',
      max_age: '3d',
    },
  },
  latest: {
    sort: '@timestamp',
    unique_key: ['vulnerability.id', 'resource.id', 'package.version'],
  },
  _meta: {
    package: {
      name: CLOUD_SECURITY_POSTURE_PACKAGE_NAME,
    },
    managed_by: 'cloud_security_posture',
    managed: true,
  },
};

As you can see the unique key is being built out of these fields:

unique_key: ['vulnerability.id', 'resource.id', 'package.version'],

The fields are:

• resource.id is currently the EC2 machine SHA.
• vulnerability.id is an id that we pull from the database like "CVE-2022-42898".
• package.version is the version of the package with a vulnerability like "7.5.0+dfsg-1" (investigating current code without the change of vulnerability.package.version).

First of all, this doesn't pinpoint the scanned EBS volumes (snapshots).
I might have multiple EBS volumes with problems but the transform will aggregate them to a single record that will point to a single EBS snapshot (I would expect to realize that I have multiple snapshots with this problem and go across them one by one).

Another problem is that different packages can have the same vulnerability.id (this is what I saw in the findings that I've received, I will post an example further on).

Also, the package.version doesn't add enough uniqueness because different packages can have the same version numbers (v1.0.0 as an example).

Now for a live example that showcases the problem with the packages and versions' uniqueness:

Notice that in this case, we have reports from different package names, but due to the transform logic, one of them will be omitted.
The same thing will also happen when I will have the same report event for the same vulnerability in different EBS volumes of the same machine.
This means that I will get a report about a certain volume, fix it in the reported EBS volume, and then next cycle I will have the same problem for the other EBS volume.

[olegsu]

Agreed to include also the ...package.name and the path of the package

• @jeniawhite to report the full path on the findings. should not block this pr.