Improve check performance by filtering it's input before parsing

[xvello]

This PR is a port of #1872 to master

What does this PR do?

The kubelet's /metrics/cadvisor payload contains statistics on all cgroups, including system slices that are of no use for the kubelet check.

The kubelet check currently filters these samples by looking of a non-empty container_name label. But this happens after we incurred the parsing, conversion and lookup costs.

On systems running a lot of system slices, this makes the kubelet check run in more than 15 seconds and use a lot of memory. One "pathological" host goes up to 40 seconds and > 1 GB memory used. 99,5 % of the kubelet payload is system slices.

This PR injects a simple text filtering component before the prometheus_client parsing logic, to remove these lines before incurring the parsing / conversion / lookup costs. For simplicity and performance, it is implemented as a list of blacklisted strings instead of regexp. If no blacklist is setup, the filtering logic is bypassed completely.

Average kubelet check run on this test payload goes from 47784ms down to 842ms. These is some CPU overhead to the filtering (with a pre-filtered payload, the check run time is ~500ms), but it is significantly amortised on even a few system slices. More info in this test notebook

On a "regular" host with 15 containers and a dozen system slices, the patch lowers the CPU usage, while keeping the memory usage constant.

We still need to optimise the processing pipeline to handle a large amount of containers, this PR does not address this.

This fix is backported on top of 6.3.2 on the datadog/agent-dev:xvello-kubelet-input-filter image (with it's jmx variant too)

Motivation

Make the kubelet check usable on hosts with lots of system slices.

Review checklist

• PR has a meaningful title or PR has the no-changelog label attached
• Feature or bugfix has tests
• Git history is clean
  - [ ] If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

[ofek]

How large might this blacklist get? You might be able to achieve increased perf if you compile it to regex then a simple compiled_re.search.

compiled_re = re.compile('|'.join(filters))

[xvello]

This is pretty drastic feature we should only use in case of obviously spammy content, not in place of the existing metric filtering. The kubelet check only needs one string, but I went with a list in case we'd need two or three (tops) in the future.