Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tag parsing optimizations #210

Merged
merged 5 commits into from
May 14, 2019

Conversation

claytono
Copy link
Contributor

@claytono claytono commented May 13, 2019

Profiling shows that for our environment, parsing tags is the single biggest steady state contributor to CPU usage. This PR replaces some calls to strings library functions to simpler implementations with the intention of reducing CPU overhead and some memory allocations.

Here is an example of the before scenario:

$ time go test -bench=BenchmarkParseDogStatsDTagsToLabels -test.benchmem -benchtime 60s
goos: darwin
goarch: amd64
pkg: github.com/prometheus/statsd_exporter
BenchmarkParseDogStatsDTagsToLabels/1_tag_w/hash-8              300000000              319 ns/op             392 B/op          6 allocs/op
BenchmarkParseDogStatsDTagsToLabels/1_tag_w/o_hash-8            300000000              318 ns/op             392 B/op          6 allocs/op
BenchmarkParseDogStatsDTagsToLabels/2_tags,_mixed_hashes-8      200000000              490 ns/op             448 B/op          9 allocs/op
BenchmarkParseDogStatsDTagsToLabels/3_long_tags-8               100000000              673 ns/op             504 B/op         12 allocs/op
BenchmarkParseDogStatsDTagsToLabels/a-z_tags-8                  20000000              5408 ns/op            3622 B/op         57 allocs/op
PASS
ok      github.com/prometheus/statsd_exporter   584.443s

And here is what it looks like afterwards:

$ time go test -bench=BenchmarkParseDogStatsDTagsToLabels -test.benchmem -benchtime 60s
goos: darwin
goarch: amd64
pkg: github.com/prometheus/statsd_exporter
BenchmarkParseDogStatsDTagsToLabels/1_tag_w/hash-8              500000000              209 ns/op             344 B/op          4 allocs/op
BenchmarkParseDogStatsDTagsToLabels/1_tag_w/o_hash-8            500000000              201 ns/op             344 B/op          4 allocs/op
BenchmarkParseDogStatsDTagsToLabels/2_tags,_mixed_hashes-8      300000000              282 ns/op             352 B/op          6 allocs/op
BenchmarkParseDogStatsDTagsToLabels/3_long_tags-8               200000000              393 ns/op             360 B/op          8 allocs/op
BenchmarkParseDogStatsDTagsToLabels/a-z_tags-8                  30000000              2871 ns/op            2374 B/op         30 allocs/op
PASS
ok      github.com/prometheus/statsd_exporter   566.397s

claytono added 4 commits May 13, 2019 12:46
Signed-off-by: Clayton O'Neill <[email protected]>
On the a-z benchmark, this brings the ns/op down from 5658 to 5533.

Signed-off-by: Clayton O'Neill <[email protected]>
This improves performance per call in the a-z case from around 5533
ns/op to 3169ns/op.  It also drops allocations per call from 57 to 31
in the a-z case.

Signed-off-by: Clayton O'Neill <[email protected]>
@claytono claytono force-pushed the tag-parsing-optimizations branch from 6967643 to 4570415 Compare May 13, 2019 16:47
@claytono claytono marked this pull request as ready for review May 13, 2019 16:47
This improves performance from 3169ns/op to 2836 ns/op and drops one
allocation per op.

Signed-off-by: Clayton O'Neill <[email protected]>
@claytono claytono force-pushed the tag-parsing-optimizations branch from 4570415 to a4faae2 Compare May 13, 2019 17:32
@matthiasr
Copy link
Contributor

Wonderful, thank you!

@matthiasr matthiasr merged commit d7eb1ed into prometheus:master May 14, 2019
matthiasr pushed a commit that referenced this pull request May 14, 2019
Signed-off-by: Matthias Rampke <[email protected]>
matthiasr pushed a commit that referenced this pull request May 15, 2019
* [CHANGE] Do not run as root in the Docker container by default ([#202](#202))
* [FEATURE] Add metric for count of events by action ([#193](#193))
* [FEATURE] Add metric for count of distinct metric names ([#200](#200))
* [FEATURE] Add UNIX socket listener support ([#199](#199))
* [FEATURE] Accept Datadog [distributions](https://docs.datadoghq.com/graphing/metrics/distributions/) ([#211](#211))
* [ENHANCEMENT] Add a health check to the Docker container ([#182](#182))
* [ENHANCEMENT] Allow inconsistent label sets ([#194](#194))
* [ENHANCEMENT] Speed up sanitization of metric names ([#197](#197))
* [ENHANCEMENT] Enable pprof endpoints ([#205](#205))
* [ENHANCEMENT] DogStatsD tag parsing is faster ([#210](#210))
* [ENHANCEMENT] Cache mapped metrics ([#198](#198))
* [BUGFIX] Fix panic if a mapping resulted in an empty name ([#192](#192))
* [BUGFIX] Ensure that there are always default quantiles if using summaries ([#212](#212))
* [BUGFIX] Prevent ingesting conflicting metric types that would make scraping fail ([#213](#213))

With #192, the count of events rejected because of negative counter increments has moved into the `statsd_exporter_events_error_total` metric, instead of being lumped in with the different kinds of successful events.

Signed-off-by: Matthias Rampke <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants