Fix handling of @timestamp and type from the JSON decoder #1421
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| } | ||
| event[k] = vstr | ||
| } else { | ||
| event[k] = v |
There was a problem hiding this comment.
Then it gets overwritten. IMHO that's fine, because keys_under_root is not the default and we usually recommend people not to use it. When people use it, it's implied that "I'm aware that everything can get overwritten".
|
Comments addressed. |
If `json.keys_under_root` and `json.overwrite_keys` are set, and the keys `@timestamp` or `type` show up in the decoded JSON, they can cause issues because the Elasticsearch output makes some assumptions about them: @timestamp has to be of type `common.Time` and `type` has to be of type `string`. The fix attempts to make use of the `@timestamp` and `type` from the JSON, but if parsing fails or the resulting values are invalid, the fields are not overwritten and an error key is added to help with troubleshooting. Fixes elastic#1378. Also hardens the `getIndex` call in libbeat, to protect against wrong types there as well.
tsg
force-pushed
the
fix_json_timestamp_overwrite
branch
from
39acc25 to
439383b
Compare
| ts, err := common.ParseTime(vstr) | ||
| if err != nil { | ||
| logp.Err("JSON: Won't overwrite @timestamp because of parsing error: %v", err) | ||
| event[jsonErrorKey] = fmt.Sprintf("@timestamp not overwritten (parse error on %s)", vstr) |
There was a problem hiding this comment.
I think it would be better to include the error and the value. Probably the error already includes the failed value.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If
json.keys_under_rootandjson.overwrite_keysare set, and the keys@timestamportypeshow up in the decoded JSON, they can cause issues becausethe Elasticsearch output makes some assumptions about them: @timestamp has to
be of type
common.Timeandtypehas to be of typestring.The fix attempts to make use of the
@timestampandtypefrom the JSON, but ifparsing fails or the resulting values are invalid, the fields are not overwritten
and an error key is added to help with troubleshooting.
Fixes #1378.
Also hardens the
getIndexcall in libbeat, to protect against wrong types thereas well.