Publish node counts grouped by service and health-check status

[petemounce]

We use statsd, and were overjoyed to see that you support that OOTB.

I had a look, and couldn't find if consul (0.4.1) publishes counts of connected agents? I'd love to be able to see:

• a count of healthy/warning/critical instances per service
• a count of present/failed/left instances per service

(where service has its name taken from the "name" element, and then sanitised to not contain any . characters)

We terminate and recycle instances in our auto scaling groups often enough that most instances don't last more than a day. Being able to see current instance count would be fantastic for us.

[armon]

@petemounce We don't have this information "easily" available. It would require doing an internal query and aggregation every interval to generate this. It may be simpler to write a small service that consumes the API to forward that information to statsd.

[petemounce]

So something that is maybe

• watch type=service that reacts to join & leave
  • does this return the name of the service?
• read /v1/catalog/service/<service> for the service-name
• netcat consul.services.<service>.nodes:<count>|g to set the node count for that service as a gauge

?

[armon]

@petemounce You may be able to do it more easily with a blocking query against /v1/health/state/any endpoint. That way you can aggregate all the healthy/warning/critical instances in a single API call.

[petemounce]

"with a blocking query" - do you mean "Blocking Queries" on https://consul.io/docs/agent/http.html ? Is there an example you could link me to of how I might do this?

[ryanuber]

@petemounce yes, a "blocking query" in Consul refers to making a specific HTTP request that will wait until some change is made. You can read more about how to perform a blocking query here under the "Blocking Queries" section (right up near the top).

[ryanuber]

@petemounce Here is an example.

First, make a query to the health endpoint and note the X-Consul-Index header:

» curl -i "localhost:8500/v1/health/state/any?pretty"
HTTP/1.1 200 OK
Content-Type: application/json
X-Consul-Index: 10
X-Consul-Knownleader: true
X-Consul-Lastcontact: 0
Date: Fri, 23 Jan 2015 18:04:41 GMT
Content-Length: 1281

[
...
    {
        "Node": "ryanuber-mbp.local",
        "CheckID": "service:redis:1",
        "Name": "Service 'redis' check",
        "Status": "passing",
        "Notes": "",
        "Output": "",
        "ServiceID": "redis",
        "ServiceName": "redis"
    },
...

Make the same query, passing in the index and an (optional) time to wait.

» curl -i "localhost:8500/v1/health/state/any?pretty&index=10&wait=1m"

The above query will "block" until the status of a health check changes, at which point a response very similar to the one above is received with the new list of services and a new X-Consul-Index, which you could use for further blocking queries.

Hope that helps!

[JesperTerkelsen]

I guess a more simple overview API or a simple command in the command line tool would be preferable ? So people can plug the aggregated status checks into nagios or datadog ?

[doublerebel]

Related: #356 /v1/health/nodes endpoint, #1164 Health service nodes filter by states

[hanshasselberg]

Thank you for reporting and participating. Since there are ways to accomplish what you asked and given that it is not simple to build in consul, I am going to close this issue.