Metricbeat json parser fails to get Docker images

[sadok-f]

Hello,

I installed Metricbeat in Ubuntu server and it sends all metrics to Eleasticsearch, however when I added docker module to see information about running containers and images, it works fine but I don't get any information about images, I checked metricbeat logs, it throws those messages:

2017-06-09T12:13:58+02:00 WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [docker.image.tags] tried to parse field [null] as object, but found a concrete value"}

• Version: 5.4.1
• Operating System: Ubuntu 16.04.2 LTS
• docker version:

Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64

Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false

• docker images:

REPOSITORY                                                             TAG                 IMAGE ID            CREATED             SIZE
alpine                                                                 3.6                 a41a7446062d        2 weeks ago         3.97 MB
alpine                                                                 latest              a41a7446062d        2 weeks ago         3.97 MB
docker.elastic.co/logstash/logstash                                    <none>              d6efb9fe9bd2        2 weeks ago         573 MB
docker.elastic.co/elasticsearch/elasticsearch                          <none>              08794661d6b0        4 weeks ago         516 MB
docker.elastic.co/kibana/kibana                                        <none>              e4fe4d5612a2        4 weeks ago         639 MB
docker.elastic.co/kibana/kibana                                        5.3.0               a21e19753b0c        8 weeks ago         679 MB
docker.elastic.co/logstash/logstash                                    5.3.0               b583a99a08a0        2 months ago        548 MB
docker.elastic.co/elasticsearch/elasticsearch                          5.3.0               ccec59a7dd84        2 months ago        206 MB

[tsg]

@sadok-f Thanks for the bug report.

I think there is an error in the template file that we generate. We have:

"tags": {
   "properties": {}
}

As a workaround, you can try to manually edit the metricbeat.template.json file, search that "tags" block and remote it. Note, however, that the template will only be applied to new indices.

[sadok-f]

@tsg Thank you for your response, I did remove the block and restart metribeat, but I still see the warning in the logs:
2017-06-12T11:32:11+02:00 WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [docker.image.tags] tried to parse field [null] as object, but found a concrete value"}

[exekias]

Hi @sadok-f, could you please dump the result of: curl --unix-socket /var/run/docker.sock http://localhost/images/json | python -m json.tool (strip anything that you don't want to share). I'm specially interested in the RepoTags field, that should be present in all items on the resulting list

[sadok-f]

Here:

[
    {
        "Containers": -1,
        "Created": 1496328202,
        "Id": "sha256:8a0824e0a178cd822eb157e1521da5801a3564994cacb286018ff78a3c8e4f03",
        "Labels": {
            "build-date": "20170510",
            "license": "GPLv2",
            "maintainer": "Elastic Docker Team <[email protected]>",
            "name": "CentOS Base Image",
            "vendor": "CentOS"
        },
        "ParentId": "",
        "RepoDigests": [
            "docker.artifactory.xxxx/telemetry/elasticsearch@*****"
        ],
        "RepoTags": [
            "docker.artifactory.xxxx/telemetry/elasticsearch:5.4.1"
        ],
        "SharedSize": -1,
        "Size": 519471729,
        "VirtualSize": 519471729
    },{
        "Containers": -1,
        "Created": 1496434536,
        "Id": "sha256:595a9e463a13702ba8425b34d6f14c08dc2985424fa32a71a2b7b55701452fc3",
        "Labels": {},
        "ParentId": "",
        "RepoDigests": [
            "php@sha256:c781ebf3a780a75948c327af9e712d16bee8de785f0263572e001e04192b43f1"
        ],
        "RepoTags": [
            "php:7.1.5-fpm-alpine"
        ],
        "SharedSize": -1,
        "Size": 68725234,
        "VirtualSize": 68725234
    },
    {
        "Containers": -1,
        "Created": 1494526614,
        "Id": "sha256:bfac4c1a30090268e38fd48dd9c4c24a80a5e26cc204ba0f2fb21ac68c1d4ef2",
        "Labels": {},
        "ParentId": "",
        "RepoDigests": [
            "php@sha256:39a75e17f607c94a7f46e3b2d29d15efbacf4d5516b7ae5914232b1f7a351f96"
        ],
        "RepoTags": [
            "php:7.1.4-fpm-alpine"
        ],
        "SharedSize": -1,
        "Size": 68587167,
        "VirtualSize": 68587167
    },
    {
        "Containers": -1,
        "Created": 1492115133,
        "Id": "sha256:20298e16e9e64eec82f8f52bd0d77eab13070b55d2d45304b09dd90931885e20",
        "Labels": {
            "org.label-schema.description": "A modern reverse-proxy",
            "org.label-schema.docker.schema-version": "1.0",
            "org.label-schema.name": "Traefik",
            "org.label-schema.url": "https://traefik.io",
            "org.label-schema.vendor": "Containous",
            "org.label-schema.version": "v1.2.3"
        },
        "ParentId": "",
        "RepoDigests": [
            "traefik@sha256:9e3c89bb72a696fe1908281dc4e99fe9ad827ff7c13776da47d031f25ad963d2"
        ],
        "RepoTags": [
            "traefik:latest"
        ],
        "SharedSize": -1,
        "Size": 41757774,
        "VirtualSize": 41757774
    }
]

We're using Artifactory as Docker registry

[tsg]

@sadok-f Note that you have to delete the existing template (or set output.elasticsearch.template.overwrite=true in the config file). You can check if the new mapping was applied by calling GET /filebeat-<data>/ in the Kibana Console.

[tsg]

@sadok-f I could reproduce the error on my computer and workaround it by following these steps:

• Stop metricbeat
• Make sure the template and existing indices are deleted, to start fresh. In Kibana Console:

DELETE /*beat-*
DELETE /_template/*

• Apply this patch to the metricbeat.template.json. Note that this asumes ES 5.x. If you have another version, you need to edit, for example: metricbeat.template-es2x.json

diff --git a/metricbeat/metricbeat.template.json b/metricbeat/metricbeat.template.json
index 3be306fc3..1e6e933bb 100644
--- a/metricbeat/metricbeat.template.json
+++ b/metricbeat/metricbeat.template.json
@@ -786,9 +786,6 @@
                 "status": {
                   "ignore_above": 1024,
                   "type": "keyword"
-                },
-                "tags": {
-                  "properties": {}
                 }
               }
             },
@@ -910,9 +907,6 @@
                       "type": "long"
                     }
                   }
-                },
-                "tags": {
-                  "properties": {}
                 }
               }
             },

• Start metricbeat

A proper fix was merged already in #4488, so I'm closing this for now. Thanks a lot for reporting and feel free to reopen if you still think there's another issue.

[sadok-f]

@tsg Many thanks !
Works now.
Thank you again!