Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using stop in a match rule does not stop metrics from being processed afterwards #246

Closed
poblahblahblah opened this issue Jan 20, 2017 · 3 comments
Labels

Comments

@poblahblahblah
Copy link

poblahblahblah commented Jan 20, 2017

Hello,

We have a number of rules that look like this:

# sin_host_000-049
match ^sin\.foobar\.sin_1a_host0[0-4][0-9]\..+ send to graphite001:d stop;

# sin_host_050-099
match ^sin\.foobar\.sin_1a_host0[5-9][0-9]\..+ send to graphite005:c stop;

We also have a rule at the very end to catch metrics which we aren't specifically sending anywhere:

# push all the rest to graphite004:d
match * send to graphite004:d stop;

What I am seeing is that the stop in all of the rules is not being honored so we end up slamming our graphite004:d carbon-cache instance. For the ruleset labeled sin_host_000-049 (and others) I have verified that carbon-c-relay is sending metrics to both graphite001:d and the default graphite004:d.

When we remove the default rule things go back to normal, as expected. carbon-cache's creates.log file created 25243 whisper files in a single minute at one point when we had this "default" backend in place.

edit: I forgot to mention OS and carbon-c-relay versions.

OS: Ubuntu 14.04 x86_64
carbon-c-relay: carbon-c-relay v2.5 (4616ad)

@poblahblahblah
Copy link
Author

Interestingly enough, when I have a decent amount of match rules (69 match rules in the config) I see this behavior, but when I have very few rules (2) this does not happen.

@grobian grobian added the bug label Jan 21, 2017
@grobian
Copy link
Owner

grobian commented Jan 21, 2017

this would be the optimiser kicking in and probably screwing it up

grobian added a commit that referenced this issue Jan 22, 2017
It may be useful to control whether or not the optimiser runs.  For
something like issue #246 it would allow a workaround, but in general
the performance could also be influenced by enabling/disabling the
optimiser.
grobian added a commit that referenced this issue Jan 22, 2017
In issue #246, the optimiser created a match group, followed by a
matchall rule.  The stop from the match group did not cause an abort of
processing any further rules /after/ the group, which is wrong
behaviour.  With this commit that behaviour is fixed.
@grobian
Copy link
Owner

grobian commented Jan 22, 2017

Got it, the optimiser wasn't the culprit, the group execution was.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants