Fix issues found in integrations after testing with logsdb

[jsoriano]

Proposed commit message

After testing with logsdb we identified some issues in packages, some of these issues seem legit.
We address them here. They were not detected before because for testing with logsdb we are:

• Testing with higher versions of the stack, what can include more fields from the agent, and ecs@mappings.
• Testing with synthetic source mode enabled in many more data streams.

Issues solved:

• Addressed fields in transforms that were not defined.
• Add definition for new fields added by the agent or other components of the stack.
• Skipped some tests that cannot be addressed now.

A new daily job is added to test with logsdb.

elastic-package changes checklist

• Fix matching of leaf fields for objects elastic-package#2054
• Add ecs@mappings multifields to all fields, not only external ones elastic-package#2055
• Misleading implementation of mappings of nested object subfields package-spec#784
• Fix validation of subfields of flattened objects elastic-package#2088
• Relax validation of subobject leaves for flattened fields elastic-package#2090

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Modify pipelines to reproduce issues.
• Remove code to hardcode stack version and logsdb. This PR will be closed.
• Revert upgrade of elastic-package. This PR will be closed.
• Enable creation of issues in LogsDB daily #11047
• Ensure we can distinguish automatic issues created for the logsdb job.
• Why are the changes needed in network_traffic.dns and ti_mandiant_advantage.threat_intelligence? https://github.com/elastic/integrations/pull/10919/files#r1735806766 Should be fixed by Fix matching of leaf fields for objects elastic-package#2054.
• Issues in github and tanium should be fixed by Misleading implementation of mappings of nested object subfields package-spec#784, as discussed in Add ecs@mappings multifields to all fields, not only external ones elastic-package#2055 (comment).
• Add daily build with logsdb enabled #11001
• Fix missing field in transforms of ti packages #11008
• Bump github.com/elastic/elastic-package from 0.102.0 to 0.103.0 #11006
• Adapt definitions of nested subfields to current Fleet implementation #11016

[andrewkroh]

A new daily job is added to test with logsdb.

Consider making it possible to distinguish "logsdb" test failures in the GH issues reported by the testreporter. Like perhaps something in the title or body.

[elasticmachine]

🚀 Benchmarks report

Package proofpoint_on_demand 👍(1) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
mail	3115.26	2364.07	-751.19 (-24.11%)	💔

Package windows 👍(2) 💚(3) 💔(4)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
applocker_msi_and_script	9174.31	6060.61	-3113.7 (-33.94%)	💔
applocker_packaged_app_deployment	9009.01	7092.2	-1916.81 (-21.28%)	💔
powershell	2074.69	1631.32	-443.37 (-21.37%)	💔
sysmon_operational	2890.17	2267.57	-622.6 (-21.54%)	💔

To see the full report comment with /test benchmark fullreport

[jsoriano]

@elastic/sec-linux-platform I have found these many new fields in the dns data stream when enabling logsdb. It seems weird because I don't see them in the 8.16 daily builds https://buildkite.com/elastic/integrations/builds/15161.

Do you know where they can come from?

This can be tested with a stack started like this:

elastic-package stack up -v --version 8.16.0-SNAPSHOT -d -U stack.logsdb_enabled=true

[jsoriano]

There is a similar issue with ti_mandiant_advantage package, it is failing for some undefined fields, but this issue doesn't happen in daily builds 🤔

[jsoriano]

For the case of ti_mandiant_advantage, the failures happen when the format of the documents look like this:

    "mandiant.threat_intelligence.ioc.attributed_associations.id": [
      "malware--6c9e3c50-490d-5a8f-8ed6-56510a62055b",
      "threat-actor--6ca32cd4-0c60-5f0b-91fb-e6e590f1f10b"
    ],
    "mandiant.threat_intelligence.ioc.attributed_associations.name": [
      "IHSBACKCONNECT",
      "UNC961"
    ],
    "mandiant.threat_intelligence.ioc.attributed_associations.type": [
      "malware",
      "threat-actor"
    ],

They pass the tests if they are formatted like this:

    "mandiant": {
      "threat_intelligence": {
        "ioc": {
          "attributed_associations": [
            {
              "id": "threat-actor--6ca32cd4-0c60-5f0b-91fb-e6e590f1f10b",
              "name": "UNC961",
              "type": "threat-actor"
            },
            {
              "id": "malware--6c9e3c50-490d-5a8f-8ed6-56510a62055b",
              "name": "IHSBACKCONNECT",
              "type": "malware"
            }
          ],

Looks like an issue with validation, because the field names are the same at the end.

[jsoriano]

elastic/elastic-package#2054 may be needed, but then I wonder why tests are passing for the second format (without synthetic source).

[jsoriano]

Confirmed, it is the same issue in network_traffic, ti_mandiant_advantage and possibly others like mongodb_atlas.

Something like elastic/elastic-package#2054 is required, testing this patch now.

[jsoriano]

An additional change is needed to avoid the different validation results with and without logsdb for these cases elastic/elastic-package#2090

[jsoriano]

Changes in ti packages moved to #11008.

[jsoriano]

Changes for nested fields moved to #11016.

[jsoriano]

These changes should not be needed as these are subfields of a flattened object, fixing this in elastic/elastic-package#2088.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 71d60a5

Failed CI Steps

• Check integrations claroty_ctd
• Check integrations cockroachdb
• Check integrations cyberarkpas
• Check integrations darktrace
• Check integrations jumpcloud
• Check integrations mongodb_atlas
• Check integrations mysql
• Check integrations snyk
• Check integrations vsphere

History

• 💔 Build #15570 failed 246ac92
• 💔 Build #15523 failed 5ccf709
• 💔 Build #15483 failed b995f9c
• 💔 Build #15472 failed 59cf4d4

cc @jsoriano

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[jsoriano]

All changes attempting to solve discrepancies in tests have been merged. The pending issues are already present when logsdb is not used. Closing this.