Custom tagging/source on logs sent to Syslog (feature)

[sshipway]

We have a number of Docker hosts that may be sending logs to Syslog via logspout. These logs appear in syslog with the Docker host IP as the source. However, we are only interested in the cluster source, not the specific host.

It would be useful to have an optional env var that can be set when starting the Logspout container, whereby you can specify the Source to use in the Syslog feed (and/or a tag to prepend to the syslog entry). In this way, the logs would arrive at syslog with a common Source of (eg) 'cluster A' rather than one of the various IP addresses of the cluster members.

Also, being able to use the Container name or ID to prepend the log entry would help in analysis where the central log server gets thousands of such logs a second.

As Docker infrastructures grow in size, this could be come a very useful feature to have available.

[alackmann]

Would love to see this as well. The Docker Cloud service (formerly Tutum) seems to set the host as the container name, which when you're logging staging/production into Loggly, results in the same hostname and an inability to tell environements apart.

[sshipway]

I've been exploring the code; although it's written in GO and there's little to no documentation, it is possible that setting the SYSLOG_HOSTNAME environment variable will do this. I'm running some tests. If it does, then SYSLOG_PRIORITY, SYSLOG_TAG, and possible SYSLOG_STRUCTURED_DATA will be able to work as well...

[alackmann]

tutum/syslogger suggests this can be done - but I've tried passing the host in as an environment variable and it doesn't seem to take effect.

But I agree - it should be feasible to pass in tags using a similar method.

[sshipway]

Tests seem to show that SYSLOG_TAG defaults to the container name (which is good) and SYSLOG_HOSTNAME defaults to the container ID (which is bad). It does not affect the syslog source as sent by the Syslog protocol, but it does change the syslog message:

1457064896 2016 Mar  4 17:14:56 +13:00 10.10.25.249: 1: 2016-03-04T04:14:56Z Rancher: rancher-agent 30049 - [] time="2016-03-04T04:14:56Z" level="info" msg="Assigning IP [10.42.47.213/16], ContainerId [81f1b90ff0bce730f1104a28d28db07a5a48c3c47273c868c21169b3c91f0e0c], Pid [30048]"

In this example , the SYSLOG_HOSTNAME was set to "Rancher:" , and the container name is "rancher-agent".

The source is still 10.10.25.149 (the IP address of the Docker host), though. Not ideal, but closer.

[deanefrati]

I ran into a similar requirement. We run containers on mesos/marathon so container IDs and names are not very meaningful so wanted to add a set of arbitrary tags to each container so that we can later do some grouping and parsing (we use ELK as a central log repository).

We we ended up doing is use the SYSLOG_TAG env variable on each container and added the tags we want as key value pairs for example:
docker run -d -e "SYSLOG_TAG=key1:val1,key2:val2,key_n:val_n" our_docker_app

We then ran logspout like this:
docker run --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_TAG={{.Container.Config.Env}} glirlabs/logspout syslog+tcp://IP:PORT

This ended up outputting a message that looked like the following:
"<14>1 2016-03-18T15:57:33Z feb5540feb8c [SYSLOG_TAG=tenant:nextgen,role:elasticsearch,app:master,source:logspuot,csp_oss_type:logs,csp_oss_codec:elasticsearch] 4650 - - I am up..",

We then parse SYSLOG_TAG with a kv filter in logstash.

Hope this helps someone.

[timbunce]

In case it helps someone...
We're using Kubernetes and identify differently clusters with what we call the 'cluster tag' - a single word like 'red', 'pink', 'orange'. We put that into a ConfigMap and then pass it into logspout as the SYSLOG_STRUCTURED_DATA env var:

          env:
            - name: SYSLOG_HOSTNAME
              value: '{{ index .Container.Config.Labels "io.kubernetes.pod.name" }}'
            - name: SYSLOG_TAG
              value: '{{ index .Container.Config.Labels "io.kubernetes.container.name" }}'
            - name: SYSLOG_STRUCTURED_DATA
              # we might want to use SYSLOG_STRUCTURED_DATA in a richer way than this later
              # but this will do for now and we can make logstash smarter later
              valueFrom:
                configMapKeyRef:
                  name: tlclusterconfig
                  key: cluster-tag

[derks]

Apologies if this is redundant, however is it possible to get the hostname of the parent host running the container? I am attempting to use PaperTrail, and having the Container ID isn't really useful. I'd like to set SYSLOG_HOSTNAME to the hostname of the parent docker host, so that in the logs you'd end up with:

• [DATE] [DOCKER_HOST_HOSTNAME] [DOCKER_CONTAINER_NAME] ...

I need this dynamically, so wondering if something similar to {{.Container.Config.Hostname}} exists for the docker host's hostname so that I can grab it on the go?

[mattatcha]

@derks Try configuring the hostname for logspout (using dockers --hostname flag) and the set SYSLOG_HOSTNAME to {{.Hostname}}.

[itkroplis]

I ran into a similar requirement. We run containers on mesos/marathon so container IDs and names are not very meaningful so wanted to add a set of arbitrary tags to each container so that we can later do some grouping and parsing (we use ELK as a central log repository).

We we ended up doing is use the SYSLOG_TAG env variable on each container and added the tags we want as key value pairs for example:
docker run -d -e "SYSLOG_TAG=key1:val1,key2:val2,key_n:val_n" our_docker_app

We then ran logspout like this:
docker run --volume=/var/run/docker.sock:/var/run/docker.sock -e SYSLOG_TAG={{.Container.Config.Env}} glirlabs/logspout syslog+tcp://IP:PORT

This ended up outputting a message that looked like the following:
"<14>1 2016-03-18T15:57:33Z feb5540feb8c [SYSLOG_TAG=tenant:nextgen,role:elasticsearch,app:master,source:logspuot,csp_oss_type:logs,csp_oss_codec:elasticsearch] 4650 - - I am up..",

We then parse SYSLOG_TAG with a kv filter in logstash.

Hope this helps someone.

=================

Can get a real example, step by step? For example, portainer docker?
sequential examples of commands?