fix: Graylog plugin TLS support and message format

[alespour]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

Resolves #9772
Resolves #9773

This PR

• fixes JSON output message where non-standard fields should be prefixed with _. New config option name_field_no_prefix can be used to use old format.
• adds TLS support for the plugin. Plugin config is extended with "standard" TLS config section like many other plugins have.

  ## Endpoints for your graylog instances.
  servers = ["udp://127.0.0.1:12201"]

  ## Connection timeout.
  # timeout = "5s"

  ## The field to use as the GELF short_message, if unset the static string
  ## "telegraf" will be used.
  ##   example: short_message_field = "message"
  # short_message_field = ""

  ## According to GELF payload specification, additional fields names must be prefixed
  ## with an underscore. Previous versions did not prefix custom field 'name' with underscore.
  ## Set to true for backward compatibility.
  # name_field_no_prefix = false

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• static_linux_amd64.tar.gz

• linux_mipsel.tar.gz

• mipsel.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• linux_i386.tar.gzi386.rpm

• freebsd_i386.tar.gz

• i386.deb

• linux_mips.tar.gz

• mips.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• windows_amd64.zip

• windows_i386.zip

• darwin_amd64.tar.gz

[srebhan]

@alespour thanks for caring about this output! The changes look reasonable, I've only some questions about the serialization where you change name to _name: Was this working before? Why do you require this change? Will it break existing setups?

A more general comment: The whole plugin looks similar to the socket_writer output with a special serialization format. If this is true, my question is, why not simply implement a serialzer plugin and use it with socket_writer?

[srebhan]

Regarding the serialization, I think you misunderstood my proposal. I'm proposing you add a new serializer under plugins/serializers/graylog and implement your serializer there. A serializer has to implement this interface and takes a metric (or a batch of metrics) and outputs []bytes. This way you can do exactly what you are currently doing in serialize() with the only difference that you output a byte stream. Would that be possible?

[alespour]

@srebhan Sorry I have misread your proposal.

Anyway, I do not think that socket_writer together with special serializer could work like graylog output plugin now. serialize function encodes the payload (ie. metric to JSON), but then there are 3 transport encoding scenarios that the plugin handles:

• UDP non-chunked - OK
• UDP chunked (proprietary) - NOT possible with serializer interface, it returns []byte, but [][]byte would be needed here
• TCP - OK

[srebhan]

Hmmm we could extend the serializer interface. @reimda, @powersj or @sspaink do you think it's worth thinking about an extension of the parser interface? We would need SerializeChunked(metric []telegraf.Metrics) ([][]byte, error)...

[srebhan]

Looks good to me. Let's follow-up on the serializer part. If you find some time to write a PoC that would be helpful...

[telegraf-tiger]

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_armv7.tar.gz
armhf.deb	i386.rpm	freebsd_i386.tar.gz
i386.deb	ppc64le.rpm	linux_amd64.tar.gz
mips.deb	s390x.rpm	linux_arm64.tar.gz
mipsel.deb	x86_64.rpm	linux_armel.tar.gz
ppc64el.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me.

[1tft]

Graylog output plugin does not handle default gelf field "full_message" as default field. It treates it as custom field and adds "_" at beginning. Thats not correct, like "short_message" field, also full_message field must not start with underscore (see https://docs.graylog.org/en/4.0/pages/gelf.html). Please can you fix this?

[Hipska]

@1tft can you create a separate issue for this? This comment will get lost in the haystack otherwise. Thanks!

[1tft]

Thank you Hipska for advice. Yesterday I reported this issue to InfluxDB support (@alespour), so I hope he will do that...