network_traffic using invalid ECS values

[jsoriano]

[0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)

Part of #3016.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I will do this after #3157 is merged.

[efd6]

This was fixed by #3343.

[jsoriano]

@efd6 I have updated the PR testing this and I see that there are still several failures in network_traffic, could you please take another look?
https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-3017/6/tests

[efd6]

It looks like these come from the beats, though it's difficult to see where since it's not coming from packetbeat and the report doesn't give any context because they are system tests. I'll send a fix to auditbeat and filebeat that are writing that value in, but will also try to replay the equivalent of https://github.com/elastic/integrations/pull/3343/files#diff-62eb4ec7064da37d476f9421d1bc3c862d8bf2f584b16a22de5b74a1f36b6f55 on each.

The sample events don't show this value, so I don't know what is going on there. This is explained in the next comment.

[efd6]

This does not fail locally (8.2.1-SNAPSHOT). The 'network_traffic' term was removed from packetbeat in elastic/beats#20556 which was not backported to 7.x, so presumably the tests that jenkins are running are on a version prior to that — I have confirmed that the tests fail on 7.17.0.

@andrewkroh What is your position on the fix for this since the removal was not backported to 7.x and the change is marked as breaking?

[jsoriano]

@efd6 an option for the integration side would be to add a processor in the ingest pipeline to fix the wrong values from old versions of Beats. This "compatibility layer" wouldn't require any change in Beats.

[efd6]

Yes, that was what I was suggesting above.

[jsoriano]

Ah ok, perfect then 🙂 I would go on this direction, without waiting for fixes in beats.

[efd6]

I'll prepare the change, but I will wait for Andrew.

[andrewkroh]

+1 for using a processor to correct the data coming from 7.x. Please put some kind of indication in the pipeline (comment or even a conditional on agent.version) that the processor is only used on 7.x data. This way we have some idea that it can be removed when we no longer support 7.x in the integration.

[efd6]

I have a comment, l'll add the conditional.

[efd6]

@jsoriano This should be good now. Please ping me if there are still issues with this integration.

[jsoriano]

@efd6 thanks a lot for your work here, the network_traffic integration is almost clean now :)

But there are still some system tests failing, it seems that network_traffic is still being used as event.category. Can you also take a look to that?

Find them here: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations%2FPR-3017/detail/PR-3017/7/tests

one or more errors found in documents stored in logs-network_traffic.thrift-ep data stream: [0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)
one or more errors found in documents stored in logs-network_traffic.thrift-ep data stream: [0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)
one or more errors found in documents stored in logs-network_traffic.tls-ep data stream: [0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)
one or more errors found in documents stored in logs-network_traffic.tls-ep data stream: [0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)

[efd6]

@jsoriano Sorry about that. I missed those two.