Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat module sophos, add null safety check and support for an additional timestamp field. #27834

Merged
merged 4 commits into from
Sep 21, 2021

Conversation

philippkahr
Copy link
Contributor

@philippkahr philippkahr commented Sep 9, 2021

What does this PR do?

Sophos in Version 18.5 sends a field called timestamp instead of the old date and time fields. Since the pipeline does not perform any null safety checks on the set command, the pipeline fails.

Additionally, sometimes sent_pkts is not populated and a trim processor is called without ignore_missing resulting in a failure.

Why is it important?

Better pipeline error handling.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Ingest simulate for the set processor ```json POST _ingest/pipeline/_simulate { "pipeline": { "processors": [ { "set": { "field": "_temp_.time", "value": "{{sophos.xg.date}} {{sophos.xg.time}}", "if": "ctx?.sophos?.xg?.date != null && ctx?.sophos?.xg?.time != null" } }, { "set": { "field": "_temp_.time", "value": "{{sophos.xg.timestamp}}", "if": "ctx?.sophos?.xg?.timestamp != null" } } ] }, "docs": [ { "_source": { "sophos": { "xg": { "date": "2021-08-12", "time": "10:00" } } } }, { "_source": { "sophos": { "xg": { "timestamp": "2021-08-12T10:00Z" } } } }, { "_source": { "sophos": { "xg": { "date": "2021-08-12", "time": "10:00", "timestamp": "2021-08-12T10:00Z" } } } }] } ```

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 9, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 9, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-09-15T11:53:05.207+0000

  • Duration: 93 min 36 sec

  • Commit: a97bda8

Test stats 🧪

Test Results
Failed 0
Passed 8259
Skipped 1201
Total 9460

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 8259
Skipped 1201
Total 9460

@mergify
Copy link
Contributor

mergify bot commented Sep 10, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-sophos-module-date-parsing upstream/fix-sophos-module-date-parsing
git merge upstream/master
git push upstream fix-sophos-module-date-parsing

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 13, 2021
@adriansr adriansr added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify bug review labels Sep 20, 2021
@marc-gr marc-gr added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Sep 20, 2021
@marc-gr marc-gr requested a review from adriansr September 20, 2021 14:38
@adriansr adriansr merged commit 0741d0a into elastic:master Sep 21, 2021
mergify bot pushed a commit that referenced this pull request Sep 21, 2021
…tional timestamp field. (#27834)

Sophos Version 18.5 sends sophos.xg.timestamp field.

Closes #27833

Co-authored-by: Adrian Serrano <[email protected]>
(cherry picked from commit 0741d0a)
mergify bot pushed a commit that referenced this pull request Sep 21, 2021
…tional timestamp field. (#27834)

Sophos Version 18.5 sends sophos.xg.timestamp field.

Closes #27833

Co-authored-by: Adrian Serrano <[email protected]>
(cherry picked from commit 0741d0a)
adriansr added a commit that referenced this pull request Sep 23, 2021
…tional timestamp field. (#27834) (#28019)

Sophos Version 18.5 sends sophos.xg.timestamp field.

Closes #27833

(cherry picked from commit 0741d0a)

Co-authored-by: Philipp Kahr <[email protected]>
Co-authored-by: Adrian Serrano <[email protected]>
adriansr added a commit that referenced this pull request Sep 23, 2021
…tional timestamp field. (#27834) (#28020)

Sophos Version 18.5 sends sophos.xg.timestamp field.

Closes #27833

(cherry picked from commit 0741d0a)

Co-authored-by: Philipp Kahr <[email protected]>
Co-authored-by: Adrian Serrano <[email protected]>
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
…tional timestamp field. (elastic#27834)

Sophos Version 18.5 sends sophos.xg.timestamp field.

Closes elastic#27833

Co-authored-by: Adrian Serrano <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify bug needs_integration_sync Changes in this PR need synced to elastic/integrations. review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Filebeat module sophos issues with timestamp and sent_pkts
5 participants