Add tenant_id option to docker log driver

[kkost]

Describe the solution you'd like
Promtail offers a way to set the tenant_id to configure the X-Scope-OrgID header when pushing logs to Loki. There should be a way to set this via a --log-opt when configuring the docker driver

Describe alternatives you've considered
Switching from the docker log driver to adding a promtil sidecar to all of my services..!

[cyriltovena]

I know this difficult to get the file config inside the docker plugin but if you manage to do it, the pipeline stages can help you here.

https://github.com/grafana/loki/blob/master/docs/clients/promtail/pipelines.md

The trouble is really to get the file accessible to the plugin.

[kkost]

That's not a particularly manageable workaround (I'd have to build separate plugins for each of my environments?). Also this isn't terribly clear - I'd need to have an "action" stage with the tenant configured as described here I think I get, but how is this config file even referenced (what do I name it?)? (I see a pipeline-example.yaml in here but that doesn't tell me much)

[cyriltovena]

That's not a particularly manageable workaround (I'd have to build separate plugins for each of my environments?).

No there is a way to access the host filesystem for a docker plugin.

You should be able to drop the file into /var/lib/docker/plugins/cb2157e1c1d9cf63f656d0ef4937f789a939b0545da056322d14409c319d6fb5/rootfs/, like config.yaml.

Then use a relative path in the --log-opt loki-pipeline-stage-file=config.yaml

The cb2157e1c1d9cf63f656d0ef4937f789a939b0545da056322d14409c319d6fb5 is the full plugin id.

To find the plugin id, you can list plugin:

docker plugin ls
ID                  NAME                                                                        DESCRIPTION           ENABLED
cb2157e1c1d9        grafana/loki-docker-driver:aggregation-592f305-WIP                          Loki Logging Driver   false

Then inspect using the short id to get the full id.

docker plugin inspect cb2157e1c1d9
[
    {
     ...
        "Id": "cb2157e1c1d9cf63f656d0ef4937f789a939b0545da056322d14409c319d6fb5",
        "Name": "grafana/loki-docker-driver:aggregation-592f305-WIP",
     ...
        }
    }
]

As an example of configuration:

pipeline_stages:
  - match:
      selector: '{app="api"}'
      stages:
        - tenant:
            value: "team-api"
  - match:
      selector: '{app="frontend"}'
      stages:
        - tenant:
            value: "team-frontend"

This would send a different tenant for two team based on the stream received. So you actually need a single file for all your tenant configuration.

If you have a single tenant you can still use this example:

pipeline_stages:
  - tenant:
      value: "my-tenant-id"

This would send all logs with my-tenant-id to Loki.

[cyriltovena]

I'm not saying we won't do your request, but I'm giving you a good workaround and pipeline stages have more to offer also, you can parse and add as a label your log level or any label that you find relevant for your log.

Since label are indexed this would make your search more easier.

[kkost]

I'm not saying we won't do your request

Thanks! Though the workaround is technically possible for me, I'd still end up writing temporary automation to build the config files and place it on the hosts (my list of tenants is not necessarily static). I will manage with the single default tenant for now, and migrate when this is ready.