[checks] Service checks should include user-configured tags

[wang-arthur]

The HTTP, TCP, and riak service checks should include user-configured tags.

Currently, the tags for these service checks are not configurable:
https://github.com/DataDog/dd-agent/blob/master/checks.d/http_check.py#L255
https://github.com/DataDog/dd-agent/blob/master/checks.d/tcp_check.py#L166-L167
https://github.com/DataDog/dd-agent/blob/master/checks.d/riak.py#L63

[VynceMontgomery]

riak service check also ignores tags; would be nice if it didn't. Not sure if that can be part of this issue or if it needs to be a separate one.
.

[wang-arthur]

Thanks @VynceMontgomery! I've updated this ticket to include riak.

[conorbranagan]

There's a current limitation in the backend on adding custom tags due to the grammar we use for service-check based monitors. Because we have implicit grouping versus explicit we will always pick the lowest granularity of grouping and alert per that group. For example, if you submit with with {host:val,port:val2,db:val3} tags, then we'll auto group results with {host,port,db}.

So this means if you add custom tags with the key:val format, they will throw off the simplest grouping.

The grammar allows for explicit grouping but up until now you did not define grouping unless you were using a multi-alert.

We're working on changing this by either (a) adding explicit grouping when you create monitors in the UI (we'll migrate existing), (b) having the alerting engine have a "known" list of groupings for non-custom checks and custom checks will have explicit grouping.

All in all, the goal is to get this working with HTTP and TCP checks first because that seems to be the most often requested case. Ideally we'll come up with a general way that can apply to every check.

[VynceMontgomery]

As a user I can tell you that I have no idea what that last comment means.

[conorbranagan]

@VynceMontgomery In short, we can't support custom tags right now due to a limitation on the backend and not in the Agent itself. We're starting work to remove this limitation and once that's changed, we'll be able to send custom tags in the Agent without trouble.

[conorbranagan]

@wang-arthur @LeoCavaille @remh Necessary changes have been made to the alerting engine so that we can support custom tags from our Agent checks. Ping me if you need more info.

[yannmh]

Partially done for a few checks: HTTP, TCP checks (see #1642).
Let's finish this for the 5.5.0

[yannmh]

Last one Riak.
Assigning you @garnermccloud, as you are working on it.