[Automatic Import] Fix missing ecs mappings #209057
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bhapas
added
release_note:fix
backport:prev-major
|
Pinging @elastic/security-scalability (Team:Security-Scalability) |
P1llus
approved these changes
Jan 31, 2025
|
Starting backport for target branches: 8.16, 8.17, 8.18, 8.x https://github.com/elastic/kibana/actions/runs/13071476437 |
💛 Build succeeded, but was flaky
Failed CI StepsTest FailuresMetrics [docs]
cc @bhapas |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2025
## Summary
Fix missing ecs mappings.
## Details
If the input is of a multilevel json where there were no ecs mappings
identified on first level, For ex:
```
{
okta_test: {
audit: {
actor: [Object],
authenticationContext: [Object],
client: [Object],
debugContext: [Object],
displayMessage: [Object],
eventType: [Object],
legacyEventType: null,
outcome: [Object],
published: [Object],
request: [Object],
securityContext: [Object],
severity: [Object],
target: null,
transaction: [Object],
uuid: [Object],
version: null
}
}
}
```
Then there is no `target` field identified and an iteration is triggered
to find the ecs mappings in the next level. Since there is no `target`
identified the logic exited and further levels were skipped in
identifying the mappings.
This PR fixes it and skips handling of Arrays since `rename` processor
cannot flatten arrays on its own to map searchable fields. This will
handled separately.
(cherry picked from commit 2fe3976)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2025
## Summary
Fix missing ecs mappings.
## Details
If the input is of a multilevel json where there were no ecs mappings
identified on first level, For ex:
```
{
okta_test: {
audit: {
actor: [Object],
authenticationContext: [Object],
client: [Object],
debugContext: [Object],
displayMessage: [Object],
eventType: [Object],
legacyEventType: null,
outcome: [Object],
published: [Object],
request: [Object],
securityContext: [Object],
severity: [Object],
target: null,
transaction: [Object],
uuid: [Object],
version: null
}
}
}
```
Then there is no `target` field identified and an iteration is triggered
to find the ecs mappings in the next level. Since there is no `target`
identified the logic exited and further levels were skipped in
identifying the mappings.
This PR fixes it and skips handling of Arrays since `rename` processor
cannot flatten arrays on its own to map searchable fields. This will
handled separately.
(cherry picked from commit 2fe3976)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
bhapas
added
the
backport:prev-minor
|
Starting backport for target branches: 8.16, 8.17, 8.18, 8.x, 9.0 https://github.com/elastic/kibana/actions/runs/13071904524 |
|
Skipping backport to 8.17/8.16 |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2025
## Summary
Fix missing ecs mappings.
## Details
If the input is of a multilevel json where there were no ecs mappings
identified on first level, For ex:
```
{
okta_test: {
audit: {
actor: [Object],
authenticationContext: [Object],
client: [Object],
debugContext: [Object],
displayMessage: [Object],
eventType: [Object],
legacyEventType: null,
outcome: [Object],
published: [Object],
request: [Object],
securityContext: [Object],
severity: [Object],
target: null,
transaction: [Object],
uuid: [Object],
version: null
}
}
}
```
Then there is no `target` field identified and an iteration is triggered
to find the ecs mappings in the next level. Since there is no `target`
identified the logic exited and further levels were skipped in
identifying the mappings.
This PR fixes it and skips handling of Arrays since `rename` processor
cannot flatten arrays on its own to map searchable fields. This will
handled separately.
(cherry picked from commit 2fe3976)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2025
## Summary
Fix missing ecs mappings.
## Details
If the input is of a multilevel json where there were no ecs mappings
identified on first level, For ex:
```
{
okta_test: {
audit: {
actor: [Object],
authenticationContext: [Object],
client: [Object],
debugContext: [Object],
displayMessage: [Object],
eventType: [Object],
legacyEventType: null,
outcome: [Object],
published: [Object],
request: [Object],
securityContext: [Object],
severity: [Object],
target: null,
transaction: [Object],
uuid: [Object],
version: null
}
}
}
```
Then there is no `target` field identified and an iteration is triggered
to find the ecs mappings in the next level. Since there is no `target`
identified the logic exited and further levels were skipped in
identifying the mappings.
This PR fixes it and skips handling of Arrays since `rename` processor
cannot flatten arrays on its own to map searchable fields. This will
handled separately.
(cherry picked from commit 2fe3976)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jan 31, 2025
## Summary
Fix missing ecs mappings.
## Details
If the input is of a multilevel json where there were no ecs mappings
identified on first level, For ex:
```
{
okta_test: {
audit: {
actor: [Object],
authenticationContext: [Object],
client: [Object],
debugContext: [Object],
displayMessage: [Object],
eventType: [Object],
legacyEventType: null,
outcome: [Object],
published: [Object],
request: [Object],
securityContext: [Object],
severity: [Object],
target: null,
transaction: [Object],
uuid: [Object],
version: null
}
}
}
```
Then there is no `target` field identified and an iteration is triggered
to find the ecs mappings in the next level. Since there is no `target`
identified the logic exited and further levels were skipped in
identifying the mappings.
This PR fixes it and skips handling of Arrays since `rename` processor
cannot flatten arrays on its own to map searchable fields. This will
handled separately.
(cherry picked from commit 2fe3976)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jan 31, 2025
# Backport This will backport the following commits from `main` to `8.x`: - [[Automatic Import] Fix missing ecs mappings (#209057)](#209057) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-31T11:04:45Z","message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport","v9.1.0"],"title":"[Automatic Import] Fix missing ecs mappings","number":209057,"url":"https://github.com/elastic/kibana/pull/209057","mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209057","number":209057,"mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Jan 31, 2025
# Backport This will backport the following commits from `main` to `8.18`: - [[Automatic Import] Fix missing ecs mappings (#209057)](#209057) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-31T11:04:45Z","message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport","v9.1.0"],"title":"[Automatic Import] Fix missing ecs mappings","number":209057,"url":"https://github.com/elastic/kibana/pull/209057","mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209057","number":209057,"mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Jan 31, 2025
# Backport This will backport the following commits from `main` to `9.0`: - [[Automatic Import] Fix missing ecs mappings (#209057)](#209057) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Bharat Pasupula","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-31T11:04:45Z","message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport","v9.1.0"],"title":"[Automatic Import] Fix missing ecs mappings","number":209057,"url":"https://github.com/elastic/kibana/pull/209057","mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209057","number":209057,"mergeCommit":{"message":"[Automatic Import] Fix missing ecs mappings (#209057)\n\n## Summary\r\n\r\nFix missing ecs mappings.\r\n\r\n## Details\r\n\r\nIf the input is of a multilevel json where there were no ecs mappings\r\nidentified on first level, For ex:\r\n\r\n```\r\n{\r\n okta_test: {\r\n audit: {\r\n actor: [Object],\r\n authenticationContext: [Object],\r\n client: [Object],\r\n debugContext: [Object],\r\n displayMessage: [Object],\r\n eventType: [Object],\r\n legacyEventType: null,\r\n outcome: [Object],\r\n published: [Object],\r\n request: [Object],\r\n securityContext: [Object],\r\n severity: [Object],\r\n target: null,\r\n transaction: [Object],\r\n uuid: [Object],\r\n version: null\r\n }\r\n }\r\n}\r\n```\r\n\r\nThen there is no `target` field identified and an iteration is triggered\r\nto find the ecs mappings in the next level. Since there is no `target`\r\nidentified the logic exited and further levels were skipped in\r\nidentifying the mappings.\r\n\r\nThis PR fixes it and skips handling of Arrays since `rename` processor\r\ncannot flatten arrays on its own to map searchable fields. This will\r\nhandled separately.","sha":"2fe39767eeee79e7e059afe5a07d7724e80c0969"}},{"url":"https://github.com/elastic/kibana/pull/209084","number":209084,"branch":"8.18","state":"OPEN"},{"url":"https://github.com/elastic/kibana/pull/209085","number":209085,"branch":"8.x","state":"OPEN"}]}] BACKPORT--> Co-authored-by: Bharat Pasupula <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fix missing ecs mappings.
Details
If the input is of a multilevel json where there were no ecs mappings identified on first level, For ex:
Then there is no
targetfield identified and an iteration is triggered to find the ecs mappings in the next level. Since there is notargetidentified the logic exited and further levels were skipped in identifying the mappings.This PR fixes it and skips handling of Arrays since
renameprocessor cannot flatten arrays on its own to map searchable fields. This will handled separately.