Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastic-Agent CEF Integration Mapping missing event.* fields #2805

Closed
iamhowardtheduck opened this issue Mar 9, 2022 · 6 comments · Fixed by #2808
Closed

Elastic-Agent CEF Integration Mapping missing event.* fields #2805

iamhowardtheduck opened this issue Mar 9, 2022 · 6 comments · Fixed by #2808
Labels
bug Something isn't working, use only for issues Integration:cef Common Event Format (CEF)

Comments

@iamhowardtheduck
Copy link

The CEF integration for Elastic-Agent is missing several event.(kind, id, start, & end) fields. Without this mapping, conflicts occur within the logs-* data view
image

@iamhowardtheduck iamhowardtheduck added bug Something isn't working, use only for issues Team:Integrations Label for the Integrations team labels Mar 9, 2022
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@kaiyan-sheng kaiyan-sheng removed their assignment Mar 9, 2022
@andrewkroh andrewkroh added Integration:cef Common Event Format (CEF) and removed Team:Integrations Label for the Integrations team labels Mar 9, 2022
@andrewkroh
Copy link
Member

I was trying to figure out how we missed this sort of issue in so many integrations. I tracked it to this line https://github.com/elastic/elastic-package/blob/a4a0b19d768eee14582acfb58a816823110a8980/internal/fields/validate.go#L266 which skips checks on event.*. Removing that check here yields:

Run pipeline tests for the package
--- Test results for package: cef - START ---
FAILURE DETAILS:
cef/log test-cef.json:
[0] field "event.code" is undefined
[1] field "event.id" is undefined
[2] field "event.severity" is undefined
cef/log test-checkpoint.json:
[0] field "event.action" is undefined
[1] field "event.category" is undefined
[2] field "event.code" is undefined
[3] field "event.duration" is undefined
[4] field "event.kind" is undefined
[5] field "event.risk_score" is undefined
[6] field "event.severity" is undefined
cef/log test-fp-ngfw-smc.json:
[0] field "event.action" is undefined
[1] field "event.code" is undefined
[2] field "event.severity" is undefined

@andrewkroh
Copy link
Member

Relates: elastic/elastic-package#147

@andrewkroh
Copy link
Member

Also found

--- Test results for package: cef - START ---
FAILURE DETAILS:
cef/log test-cef.json:
[0] parsing field value failed: field "event.id"'s Go type, float64, does not match the expected field type: keyword (field value: 123)
[1] parsing field value failed: field "event.id"'s Go type, float64, does not match the expected field type: keyword (field value: 3457)
cef/log test-checkpoint.json:
[0] parsing field value failed: field "event.risk_score"'s Go type, string, does not match the expected field type: float (field value: 4)
[1] parsing field value failed: field "event.severity"'s Go type, string, does not match the expected field type: long (field value: 4)

andrewkroh added a commit to andrewkroh/integrations that referenced this issue Mar 9, 2022
Several event field mappings were missing (tests do not validate event.* as per elastic/elastic-package#147).
After adding those mappings some of the data types didn't match so I added a few convert processors.

I modified the pipeline tests to use simulated data from the Beats decode_cef processor.

Fixes: elastic#2805
@andrewkroh
Copy link
Member

PR to fix: #2808

andrewkroh added a commit that referenced this issue Mar 10, 2022
Several event field mappings were missing (tests do not validate event.* as per elastic/elastic-package#147).
After adding those mappings some of the data types didn't match so I added a few convert processors.

I modified the pipeline tests to use simulated data from the Beats decode_cef processor.

Fixes: #2805
eyalkraft pushed a commit to build-security/integrations that referenced this issue Mar 30, 2022
Several event field mappings were missing (tests do not validate event.* as per elastic/elastic-package#147).
After adding those mappings some of the data types didn't match so I added a few convert processors.

I modified the pipeline tests to use simulated data from the Beats decode_cef processor.

Fixes: elastic#2805
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working, use only for issues Integration:cef Common Event Format (CEF)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants