Support Consul service endpoints for service discovery

[johnrengelman]

Currently, confd is limited to accessing the KV store of consul. It would be very helpful if the service and nodes endpoints could also be accessed. Then things like writing a HAProxy conf file can be done by querying the healthy endpoints for a service in Consul (not sure what/if there is a similar feature in etcd).

Would there be interest in a feature like this? I may be able to get schedule some work at my job to develop it if there is interest.

The main question without looking at anything yet is how would the confd templating discriminate between normal KV and the service or node data.

[carlosdp]

Yep, I would very much like this and would like to start working on it. It makes more sense than just the kv store because those don't take advantage of Consul's health checks and such. My first thought was to have a flag to enable service endpoints and then namespacing those keys under a service_ prefix like so:

upstream web {
  {{range .service_web}}
    {{.Node.Address}}:{{.Service.Port}}
  {{end}}
}

[johnrengelman]

I got something put together last week that works pretty nice (based off the 0.6.x branch).

I didn't add a flag to control the generation, as I thought it was just something that should be supported out of the box. I went to the route of just injecting the service data into the KV using paths:

/_consul/services/<service_name>/<idx>/address
/_consul/services/<service_name>/<idx>/port

And additionally, Tags were supported by adding to the path:

/_consul/services/<service_name>/<tag>/<idx>/address
/_consul/services/<service_name>/<tag>/<idx>/port

Then in the template, I could use the normal KV methods:

{{range gets "/_consul/service/web/*/address"}}
  {{.Value}}
{{end}}

I like the structure of your example above though, but I find it might be hard to support tags and datacenters using the style.

Perhaps something that's a hybrid where, we use a method to get the service information?

upstream web {
  {{range (.service 'web')}}
    {{.Node.Address}}:{{.Service.Port}}
  {{end}}
}

Then, that could be expanded to include tags:

{{range (.service 'web' 'tag1')}}
{{end}}

[carlosdp]

Actually, I like your implementation better. No need to abstract the Consul api 😄 I'm not that familiar with gotext templating so I was just going off of the confd docs.

Yea, if you post that PR I'll defintely help!

[johnrengelman]

Here's my first crack at this: #102

[mohitarora]

Consul documentation says: Lookup consul nodes using SRV records will be available in 0.5.0

Is there any timeline around 0.5.0? I can work on it but i see PR already submitted for this.

Please let us know.

[johnrengelman]

The SRV record support is for using SRV records from consul to detect the backend nodes for confd itself, not for acquiring the consul service information and exposing it to templates. I was confused by that initially as well.

Yeah, I put #102 out there as my attempt to solve this problem.

[kelseyhightower]

See #102 but the idea is that we will add support this via custom backends.