Update integrations.md to point to official Elasticsearch implementation

[ChrsMark]

This PR updates the docs to point to the official integration for Elasticsearch which uses an official module of Beats upstream project instead of a custom community Beat.

Integration implementation issue: elastic/beats#14983

[roidelapluie]

Thank you. I think we want to hold on this until it is in a stable release ? Also, did you talk with the people behind the repository you are replacing?

Thanks

[ChrsMark]

👍 on waiting for the release. Owners of the other project will also be informed about it then! I will let you know! Thanks!

[roidelapluie]

The best thing I'd imagine is if they would agree on pointing to the official implementation in their readme so we point the community to the official version only. I hope they both produce the same output in elasticsearch.

[brian-brazil]

There is a preference for official integrations, we do need to check that it is a reasonable replacement though. Engaging with the existing integration is encouraged.

I hope they both produce the same output in elasticsearch.

This seems not to be the case, though I don't think that's a blocker as the current beat will still exist.

Having a map called "metrics" with a single entry feels a bit odd. I'm not experienced with ES, might separate name/value fields under "prometheus" make more sense? (Which incidentally would be closer to what the existing one is doing)

One question: How do you handle NaN, +Inf, and -Inf? Both integrations seem to be using JSON numbers, which don't support those. We solve this in the Prometheus HTTP APIs by using strings for all our values.

You've also a typo "configureed". As the docs are currently written I could also see users enabling TLS on the ES side, but missing that they need to update to https:// on the Prometheus side.

[ChrsMark]

Hey folks let me give an update on this:

Having a map called "metrics" with a single entry feels a bit odd. I'm not experienced with ES, might separate name/value fields under "prometheus" make more sense? (Which incidentally would be closer to what the existing one is doing)

Metricbeat had already support for collecting and storing Prometheus metrics. This could be achieved by enabling and using collector metricset of prometheus module, which periodically scrapes metrics from Prometheus endpoints. In our case remote_write is another metricset of the prometheus module and hence it was designed to follow the same storing format with collector metricset so as to have a unified way of storing Prometheus metrics with both approaches. An example of how Prometheus metrics are being transformed into Metricbeat events in order to be stored in Elasticsearch is the following:

{
    "@timestamp": "2019-03-01T08:05:34.853Z",
    "event": {
        "dataset": "prometheus.collector",
        "duration": 115000,
        "module": "prometheus"
    },
    "metricset": {
        "name": "collector",
        "period": 10000
    },
    "prometheus": {
        "labels": {
            "job": "prometheus",
            "listener_name": "http"
        },
        "metrics": {
            "net_conntrack_listener_conn_accepted_total": 3,
            "net_conntrack_listener_conn_closed_total": 0
        }
    },
    "service": {
        "address": "127.0.0.1:55555",
        "type": "prometheus"
    }
}

As we can see both metricsets put labels under prometheus.labels.* and metrics' values under prometheus.metrics.*. This is a convention that we would like to preserve across the prometheus module.

One question: How do you handle NaN, +Inf, and -Inf? Both integrations seem to be using JSON numbers, which don't support those. We solve this in the Prometheus HTTP APIs by using strings for all our values.

Currently we don't store these metrics, however it's something that could be supported in the future.

You've also a typo "configureed". As the docs are currently written I could also see users enabling TLS on the ES side, but missing that they need to update to https:// on the Prometheus side.

Docs have been updated.

Let me know if there is anything else I can address :).
ps: I can also communicate with the currently documented project's maintainers if it is needed.

[brian-brazil]

Thanks!