[system.auth] Split grok parsing, add tags/processors/preserve_original_event #3705
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
🤖 GitHub commentsTo re-run your PR in the CI, just comment with:
|
The script failed with "cannot access method/field [event] from a null def reference". And the geoip failed when the source.ip was missing.
🌐 Coverage report
|
andrewkroh
added
Team:Security-External Integrations
Team:Elastic-Agent-Data-Plane
|
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
6 tasks
efd6
reviewed
Jul 14, 2022
packages/system/data_stream/auth/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/system/data_stream/auth/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
And add a test for multline logs.
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 19, 2022
Sync the pipeline for the system.auth dataset with the Fleet integration from elastic/integrations#3705. This removes the event.type authentication_failed and authentication_success values which are not allowed as per ECS. You can use event.category: authentication and event.outcome: success/failure to query instead.
efd6
approved these changes
Jul 19, 2022
andrewkroh
added a commit
to elastic/beats
that referenced
this pull request
Jul 20, 2022
Sync the pipeline for the system.auth dataset with the Fleet integration from elastic/integrations#3705. This removes the event.type authentication_failed and authentication_success values which are not allowed as per ECS. You can use event.category: authentication and event.outcome: success/failure to query instead.
mergify bot
pushed a commit
to elastic/beats
that referenced
this pull request
Jul 20, 2022
Sync the pipeline for the system.auth dataset with the Fleet integration from elastic/integrations#3705. This removes the event.type authentication_failed and authentication_success values which are not allowed as per ECS. You can use event.category: authentication and event.outcome: success/failure to query instead. (cherry picked from commit 475dd7e)
andrewkroh
added a commit
to elastic/beats
that referenced
this pull request
Jul 26, 2022
…ation (#32422) * system.auth - sync pipeline with Fleet integration (#32360) Sync the pipeline for the system.auth dataset with the Fleet integration from elastic/integrations#3705. This removes the event.type authentication_failed and authentication_success values which are not allowed as per ECS. You can use event.category: authentication and event.outcome: success/failure to query instead. (cherry picked from commit 475dd7e) * Update go-ucfg to v0.8.6 The module tests were failing because the empty tags array was being discarded resulting in the tags value being treated as `nil`. And this was not being accepted as a valid type for the inList function. The cause was elastic/go-ucfg#188. * Fix tests affected by go-ucfg upgrade Backport fix from 0022ea4. Co-authored-by: Andrew Kroh <[email protected]>
|
@andrewkroh Is this change getting back port to 7.17 branch for future Beats 7.17.x releases, or will it only be in 8.3.x/8.4+? Thx! |
|
The related Filebeat changes are
That covers 8.4 and 7.17. |
chrisberkhout
pushed a commit
to elastic/beats
that referenced
this pull request
Jun 1, 2023
Sync the pipeline for the system.auth dataset with the Fleet integration from elastic/integrations#3705. This removes the event.type authentication_failed and authentication_success values which are not allowed as per ECS. You can use event.category: authentication and event.outcome: success/failure to query instead.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This adds config options for tags, processors, and preserve original event to the system.auth data stream.
It improves the pipeline by splitting the main grok processor into two operations. First it groks the header common to all messages. Then the second grok extracts information from a specific set of messages. Previously all of the patterns in the single grok shared the same header so the the header was being parsed multiple times while trying to find a matching pattern. Additionally the patterns where not anchored so those were added.
I added a convert processor to check the
source.ipfor validity.I added an
ifcondition to the script processor to preventcannot access method/field [event] from a null def referenceerrors that were being ignored withignore_failure: true.Checklist
changelog.ymlfile.Related issues
Screenshots
Before:
After: