[Response Ops][Alerting] Adding ignore_malformed to .alerts-* index template settings
#163414
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ymao1
added
Feature:Alerting
release_note:skip
|
Pinging @elastic/response-ops (Team:ResponseOps) |
…-ref HEAD~1..HEAD --fix'
…malformed fix test for geo_point
ymao1
added
release_note:skip
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @ymao1 |
vitaliidm
approved these changes
Aug 9, 2023
1 task
bryce-b
pushed a commit
to bryce-b/kibana
that referenced
this pull request
Aug 9, 2023
…ex template settings (elastic#163414) Resolves elastic#161465 ## Summary Adds `ignore_malformed: true` to alerts index template settings. This ignores malformed content globally across all allowed mapping types. For existing alerts as data indices, the new setting is not applied directly to the existing concrete indices but will be applied whenever the alias rolls over and a new concrete index is created. ## Verify - Verify that after upgrading alerts indices created in an older version to this branch, alerts continue to be written and read as expected. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Vitalii Dmyterko <[email protected]>
vitaliidm
added a commit
that referenced
this pull request
Aug 10, 2023
…ion (#163487) ## Summary While reviewing #163414, I have noticed, that `geo_point` type still gets validated in `computeIsEcsCompliant`, that could lead to removing some of the complex `geo_point` type representations, notably object like ones, for example ```JSON { "type": "Point", "coordinates": [-88.34, 20.12], } ``` In this PR, I completely removing validation for this field type (we even have functional test to verify it) With changes introduced in #163414, alert will be created even with not valid `geo_point` fields, instead of failing (changed e2e test name) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
ymao1
added a commit
to ymao1/kibana
that referenced
this pull request
Aug 10, 2023
…s-*` index template settings (elastic#163414)" This reverts commit 0c51ce6.
ymao1
added a commit
that referenced
this pull request
Aug 10, 2023
Reverting #163414 and #163487 ## Summary @pmuellr uncovered a bug in ES with `ignore_malformed` and datastreams while working on #154266 > Found what I hope is an ES bug yesterday w/data streams (DS). It doesn’t like ignore_malformed on the @timestamp field :slightly_smiling_face:. I think this is a bug since [the doc says (https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html#ignore-malformed-setting) Mapping types that don’t support the setting will ignore it if set on the index level. I think it’s understandable - the @timestamp field is a key field for DS (can be overridden) - so you’d not be surprised it’s treated specially. But … why not just ignore it in that case, like the other mapping types that are ignored. I tried overriding ignore_malformed for just that field, and it complained that I couldn’t use that option on that field! hahahahah So, we’d be left having to add ignore_malformed to every mapped field in our mappings, except for @timestamp. For the time being, I’ve removed all the ignore_malformed stuff in my AaD DS PR, when using DS, but left it when using alias/index. Unless someone knows more about this special ignored_malformed / @timestamp field / data-stream relationship, I’ll boil down a simple test case and open an issue for ES. In order to avoid having even more divergent code between serverless & serverful, we will revert this change until we can confirm a bug with ES and hopefully get a fix in.
pmuellr
added a commit
to pmuellr/kibana
that referenced
this pull request
Sep 8, 2023
Resolves elastic#161465 This is a re-do of elastic#163414, which we had to revert since data streams do not support `ignore_malformed` on the `@timestamp` field. We now specifically add `ignore_malformed: false` for that field, and then use `ignore_malformed: true` at the index level. This ignores malformed content globally across all allowed mapping types. For existing alerts as data indices, the new setting is not applied directly to the existing concrete indices but will be applied whenever the alias rolls over and a new concrete index is created. - Verify that after upgrading alerts indices created in an older version to this branch, alerts continue to be written and read as expected.
pmuellr
added a commit
that referenced
this pull request
Sep 14, 2023
Resolves #161465 This is a re-do of #163414, which we had to revert since data streams do not support `ignore_malformed` on the `@timestamp` field. We now specifically add `ignore_malformed: false` for that field, and then use `ignore_malformed: true` at the index level. This ignores malformed content globally across all allowed mapping types. For existing alerts as data indices, the new setting is not applied directly to the existing concrete indices but will be applied whenever the alias rolls over and a new concrete index is created.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #161465
Summary
Adds
ignore_malformed: trueto alerts index template settings. This ignores malformed content globally across all allowed mapping types. For existing alerts as data indices, the new setting is not applied directly to the existing concrete indices but will be applied whenever the alias rolls over and a new concrete index is created.Verify