fix(outputs/stackdriver): cumulative interval start times

[n-oden]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #7758

Cumulative metrics, when posted to google cloud monitoring (nee
Stackdriver), require a TimeInterval with a start time that corresponds
to the last time at which the counter/cumulative metric was reset to
zero, cannot overlap previous intervals, and cannot be more than 25
hours in the past:

https://cloud.google.com/monitoring/api/ref_v3/rest/v3/TimeInterval

The stackdriver output plugin had been creating StartTimes of
Thu Jan 1 00:00:01 UTC 1970 leading to bizarre behavior: stackdriver graphs
would reflect flatlined fractional values no matter what data telegraf
was sending, and direct queries to the projects.timeSeries.list API
would initially return the last handful of posted values but then
eventually return only {"unit": "not_a_unit"} for the same query,
presumably due to stackdriver having GC'ed the invalid intervals on the
backend.

So, herein:

• create a cache of start times and observed last values for all counter metrics (keyed by name, tags and field)
• reset the start time of a cache entry if the newest value is less than the previous observed value (counter reset)
• reset the start time of a cache entry if it is greater than 24 hours old
• while we're at it, use the telegraf logger rather than the base golang logger

[n-oden]

Note that as mentioned in the readme update, even with this change there is the potential for undefined behavior if one of the underlying counters resets without telegraf restarting in concert with it. Doing this "right" would probably involve caching a separate start time and the last observed value for each counter metric and resetting the start time any time the metric resets, which seems to somewhat cut against the grain of telegraf's normal near-statelessness on the output side, but if there's interest I could attempt to implement that.

[n-oden]

Actually, having thought on it a bit, the per-metric cache really seems to be the only way to do this: setting the start time to the process start time means that if the input source starts exposing a new counter >25 hours after we start, that counter will never make it to stackdriver.

So, an update, complete with unit tests.

[n-oden]

@sspaink @Hipska @powersj Could I get a look taken at this? (Apologies in advance if I've missed a step here.)

[srebhan]

I guess this would benefit from plugin state-persistence (PR #9476)?

[srebhan]

Thanks for approaching this @n-oden! While the overall approach looks good, I have some comments in the code. Please try to reduce the non-related changes (renaming of aliases, reordering of imports, etc) to an absolute minimum and submit them as separate PRs to ease review. Furthermore, do you think it's possible to leave getStackdriverTimeInterval() as is and instead determine the start and end times in a separate function?

[n-oden]

hey @srebhan glad to finally have a review for this! Your comments seem reasonable; I'll try to get them all addressed today or tomorrow.

[srebhan]

@n-oden sorry for taking so long to come to this... :-(

[n-oden]

@srebhan should be good to go, I think!

[srebhan]

@n-oden thanks for the update. I have some more comments, but we are almost there I think.

[n-oden]

@srebhan all done I think?

[srebhan]

The linter found some more... :-)

[telegraf-tiger]

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Looks like new artifacts were built from this PR.

Expand this list to get them here ! 🐯

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_i386.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	s390x.rpm	linux_amd64.tar.gz
mipsel.deb	x86_64.rpm	linux_arm64.tar.gz
ppc64el.deb		linux_armel.tar.gz
s390x.deb		linux_armhf.tar.gz
		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

[srebhan]

Looks good to me. Thanks for fixing this long-standing issue @n-oden!