event: add max eps reporting.

[AlexJF]

This PR adds periodic max eps sampling reporting. This reporting is 2-fold:

• Statsd gauges for current, max, sample_rate and max_reached.
• Log lines prompting the user to tune the MaxEPS setting if the maximum event rate was reached.

Sample output:

=== RUN   TestEventProcessorFromConf/metric_-_above_max_eps
1541500869865907000 [Warn] Max events per second reached (current=193.66/s, max=100.00/s). Some events are now being dropped (sample rate=0.52). Consider increasing MaxEPS setting.
1541500899864986000 [Warn] Max events per second reached (current=224.48/s, max=100.00/s). Some events are now being dropped (sample rate=0.45). Consider increasing MaxEPS setting.

[gbbr]

I think this resulted in being a bit over-engineered in an attempt to fit in this decorator pattern which is not natural to Go imho. The fact that a type cast is required and that the new methods added on the sampler say nothing about their use (for stats) is a good indication that this isn't the way to do it.

I'd prefer it if we didn't go with this solution. The intention here is to only report occasionally, right?

Instead, let's:

• Add a method (*maxEPSSampler).report which reports to statsd when called.
• Inside (*maxEPSSampler).Start create a new routine which calls that method on an interval:

go func() {
    for range s.reportTicker.C {
        s.report()
    }
}()

• Call reportTicker.Stop in (*maxEPSSampler).Stop

In my opinion this is the way to go. This way, instrumentation is separated out into its own method as you've intended, while still not decoupled from its source. At this stage, we don't need a reusable component. Even so, if we were to create one we should do it differently.

---

If we need a reusable component later, we could create an interface:

type Reporter interface {
    Report(client statsd.Client)
}

We could then simply add this method to the sampler (or any other component) to enable it as a candidate for interval reporting. A separate struct could use these reporters to call them on an interval. This way the implementation and intention is clear, with not much modification required.

[gbbr]

Edit: the for loop for the ticker in my example needs to be extended a bit, more along the lines of the example in https://golang.org/pkg/time/#NewTicker to exit cleanly.

[gbbr]

Thanks Alex. LGTM. Left a couple more nits but don't feel strongly about them so I'll leave it up to you.

[gbbr]

I think we can remove the reportFrequency from the arguments list, since it's not likely to be needed there, right? Then, for testing, you can keep it in the unexported constructor. It could be just a const.

[gbbr]

With the risk of becoming annoying here: can we remove all the whitespace?

[gbbr]

LGTM