in_forward: add support to configure tags for records ingested using the forward input #4455
Labels
Comments
edsiper
added a commit
that referenced
this issue
Dec 16, 2021
This patch adds a new option called 'tag_prefix' that allows to prefix the
incoming Tag set by the remote client.
Usage:
--- config file ---
[SERVICE]
flush 1
log_level info
[INPUT]
name forward
tag_prefix new.
[OUTPUT]
name stdout
match *
--- end ---
> terminal 1:
$ echo "{\"key\": \"test\"}" | fluent-cat test
> terminal 2:
$ bin/fluent-bit -i forward -p tag_prefix=new. -o stdout -f 1
...
[0] new.test: [1639626354.501658295, {"key"=>"test"}]
Signed-off-by: Eduardo Silva <[email protected]>
|
Implementation submitted on #4466 |
edsiper
added a commit
that referenced
this issue
Dec 16, 2021
This patch adds a new option called 'tag_prefix' that allows to prefix the
incoming Tag set by the remote client.
Usage:
--- config file ---
[SERVICE]
flush 1
log_level info
[INPUT]
name forward
tag_prefix new.
[OUTPUT]
name stdout
match *
--- end ---
> terminal 1:
$ echo "{\"key\": \"test\"}" | fluent-cat test
> terminal 2:
$ bin/fluent-bit -i forward -p tag_prefix=new. -o stdout -f 1
...
[0] new.test: [1639626354.501658295, {"key"=>"test"}]
Signed-off-by: Eduardo Silva <[email protected]>
|
This PR above has been tested and verified. It fulfilled our need. |
|
Note: Looks like it's not documented yet. |
edsiper
added a commit
that referenced
this issue
Jan 6, 2022
This patch adds a new option called 'tag_prefix' that allows to prefix the
incoming Tag set by the remote client.
Usage:
--- config file ---
[SERVICE]
flush 1
log_level info
[INPUT]
name forward
tag_prefix new.
[OUTPUT]
name stdout
match *
--- end ---
> terminal 1:
$ echo "{\"key\": \"test\"}" | fluent-cat test
> terminal 2:
$ bin/fluent-bit -i forward -p tag_prefix=new. -o stdout -f 1
...
[0] new.test: [1639626354.501658295, {"key"=>"test"}]
Signed-off-by: Eduardo Silva <[email protected]>
0Delta
pushed a commit
to 0Delta/fluent-bit
that referenced
this issue
Jan 20, 2022
This patch adds a new option called 'tag_prefix' that allows to prefix the
incoming Tag set by the remote client.
Usage:
--- config file ---
[SERVICE]
flush 1
log_level info
[INPUT]
name forward
tag_prefix new.
[OUTPUT]
name stdout
match *
--- end ---
> terminal 1:
$ echo "{\"key\": \"test\"}" | fluent-cat test
> terminal 2:
$ bin/fluent-bit -i forward -p tag_prefix=new. -o stdout -f 1
...
[0] new.test: [1639626354.501658295, {"key"=>"test"}]
Signed-off-by: Eduardo Silva <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently the Forward Input plugin doesn't allow overwriting or customizing Tags for log records in any way.
I'm trying to add support for a
forwardreceiver in the Ops Agent. But doing so requires that the Ops Agent be able to route records to processors/filters and outputs using Tags. It can't do that with the existing Fluent Bit Forward receiver as it doesn't allow the Ops Agent to control the Tags of records.Describe the solution you'd like
If the Forward receiver exposed a configuration field like "Tag_Prefix" (defaulted to "") that would prefix the tags from logs ingested with a configurable string - it would allow the Ops Agent to better control the data pipeline.
Describe alternatives you've considered
The Forward output plugin allows a configuration option that overwrites the Tag. We could have the same for the Input plugin.
Cons: The old tag is discarded and this is not good. It could be used for further processing, routing etc. If this is done, we should figure out where to store the old tag (push it into the message?)
Additional context
"Tags" aren't exposed by the Ops Agent to its users, and alternatives we've considered (like using rewrite_tag don't work with multiple forward inputs). Allowing an option like "Tag_Prefix" would let the Ops Agent prefix the record tags with some other string (like receiver ID) and so would let the Ops Agent continue routing and processing records.
The text was updated successfully, but these errors were encountered: