[Metricbeat] Support pulling secrets from Kubernetes/OpenShift for autodiscovery

[mikeh-elastic]

Currently autodiscovery only supports unauthenticated module metricset collection via autodiscovery hints.

As container orchestration platforms can generate passwords and usernames that are unknown to metricbeat at container launch we need a way to collect and utilize these passwords.

Kubernetes and OpenShift have a secrets api that could be allowed for metricbeat to read secrets for containers autodiscovered https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.10/#-strong-read-operations-strong--211

From an RBAC perspective we can start testing with a permissive role that allows metricbeat to read and list all secrets and move towards a role where it can only read secrets that it knows that exist through autodiscovery by name to limit access further.

Here is an example application secret api call response, note that we can have several values in the data object to reference:

{
    "apiVersion": "v1",
    "data": {
        "database-admin-password": "aFV0B0VzME1YbXZLSjgzNA==",
        "database-password": "Mjc2RkHhN1VydnNSVUxPUA==",
        "database-user": "dXWlcjFOMA=="
    },
    "kind": "Secret",
    "metadata": {
        "annotations": {
            "openshift.io/generated-by": "OpenShiftNewApp"
        },
        "creationTimestamp": "2018-06-01T19:12:38Z",
        "labels": {
            "app": "nodejs-mongodb-example",
            "template": "nodejs-mongodb-example"
        },
        "name": "nodejs-mongodb-example",
        "namespace": "apm",
        "resourceVersion": "593414",
        "selfLink": "/api/v1/namespaces/apm/secrets/nodejs-mongodb-example",
        "uid": "bf9295cd-65cf-11e8-9ec5-42010a800003"
    },
    "type": "Opaque"
}

Using the annotations autodiscover hints, and knowing the namespace of the pod, we could support something like this example to pull the secret by name and value from the correct namespace and write an autodiscover module config with username and password:

co.elastic.metrics/secret.username: secret-name.database-user
co.elastic.metrics/secret.password: secret-name.database-password

[exekias]

Thanks for opening this issue @mikeh-elastic. Beats keystore was created with this in mind, we aim to allow pluggable keystore backends, that would allow Beats to read (and even write) it's secrets to Kubernetes secrets.

The user would be able to reference them using the current variable notation. I'm thinking on something like this: ${kubernetes:namespace:secretname:var}

cc @ph

[ph]

@exekias Initially, I thought we could go away with not having to specify where the password is located and just rely on first found principle, but now I see value of prefixing the secret with the keystore name (kubernetes) to allow easier debugging, btw we have an issue open at #5832

[exekias]

I want to get this effort back to life. We should think about the security implications of allowing Metricbeat reading arbitrary secrets.

pinging @odacremolbap @ChrsMark any thoughts on how this could work?

[ChrsMark]

Hey @exekias this sounds interesting!

Thinking about this quickly I guess that it depends. A known limitation about secrets is that everyone with access to a specific namespace can create a Pod to "exploit" this secret by mounting this secret, right? My point here is that "if a I have permission to launch Metricbeat in a spesific namespace this means that I already have access to all secrets of this specific namespace" . So the lower bound is that Metricbeat should only be able to access secret resources through API at least in the same namespace which is doable using the proper RBAC?

[exekias]

That's fair, still that would mean Metricbeat can only monitor containers in the same namespace (at least if secrets are needed). When we deploy Metricbeat as a DaemonSet it's minded to allow users to monitor the whole cluster, including workloads in all namespaces.

Also, by allowing hints based autodiscover, we are allowing any user in the cluster to define how to monitor their workloads. Can we allow them to reference secrets anyhow?

One possible option is to allow users to reference secrets in their own namespace. So if I'm adding annotations for Metricbeat, I can reference some secret in the same namespace as the pod that I'm annotating. Metricbeat would need to enforce hints/autodiscover only access secrets from the same namespace as the pod it's monitoring.

[ChrsMark]

That's fair, still that would mean Metricbeat can only monitor containers in the same namespace (at least if secrets are needed). When we deploy Metricbeat as a DaemonSet it's minded to allow users to monitor the whole cluster, including workloads in all namespaces.

Also, by allowing hints based autodiscover, we are allowing any user in the cluster to define how to monitor their workloads. Can we allow them to reference secrets anyhow?

One possible option is to allow users to reference secrets in their own namespace. So if I'm adding annotations for Metricbeat, I can reference some secret in the same namespace as the pod that I'm annotating. Metricbeat would need to enforce hints/autodiscover only access secrets from the same namespace as the pod it's monitoring.

Hey @exekias , I would say that we could make an initial implementation for this if there isn't any strong objection. What do you think about transferring your above proposal into a feature-issue? I would be more than happy to work on this then.

[ChrsMark]

Giving a heads-up on this a possible solution here would be to implement a new Keystore for Kubernetes secrets which then would be passed to configuration opitons at

beats/libbeat/autodiscover/template/config.go

Line 102 in 2c70a2b

opts := []ucfg.Option{

. I will investigate this.

Maybe related to: #12597