Fix fields handling to properly batch HTTP request payloads #54
Conversation
| ] | ||
| end | ||
|
|
||
| def dump_collected_fields(log_fields) |
There was a problem hiding this comment.
this function will likely come back after we do #53
| fields = sumo_metadata['fields'] || "" | ||
| fields = extract_placeholders(fields, chunk) unless fields.nil? | ||
|
|
||
| "#{source_name}:#{source_category}:#{source_host}:#{fields}" |
There was a problem hiding this comment.
not addressed by this PR, but something we should fix: having a colon (:) in any metadata field or value would currently break this
There was a problem hiding this comment.
good point - I believe you can use ruby hashes as keys? maybe that's something we can look into though it does make it pretty complex
There was a problem hiding this comment.
LGTM. Something we should put into the backlog potentially is some basic inspecting of the fields to ensure they meet requirements. E.g. max of 20, etc. Not the focus of this PR just bringing it up. Agree with the Colon. I just did a check on my clusters and did not see anything but wondering if its worth a proper fix?
|
Yeah these are all great points, I am thinking start with this for now, and revisit our HTTP batching to account for these edge cases. |
Noticed by Paul, after adding the additional
fieldsto_sumo_metadata, we were not properly batching the HTTP request payloads in order to setX-Sumo-Fields.This PR adds handling for
fieldsfrom_sumo_metadataand adds test cases for:Some of this may conflict with #53 but we can reconcile the differences.