Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(outputs.redistimeseries): Handle string fields correctly #14060

Merged
merged 7 commits into from
Oct 12, 2023

Conversation

srebhan
Copy link
Member

@srebhan srebhan commented Oct 6, 2023

resolves #14058

Redis Timeseries do only support numeric field values. Currently, we bluntly send the values to Redis irrespective of their type and receive errors on string fields that convert non-numeric characters. This error then aborts the write and in turn to a retry which will only retrigger the error.

This PR adds a convert_string_fields option which by default tries to convert string-fields to numeric values succeeding if they e.g. contain "3.14" or "0" but failing on any non-convertible values like "3.1.2" or "foo". In the former case the value is sent to Redis while in the latter (failing) case the field is dropped but sending is continued. This resembles the current behavior as close as possible.
When setting the option to false all string fields are dropped and no attempt is made to safe them.

In the course of the PR, we also update the redis client to v9 allowing support for timeseries commands and adding unit tests including a "testcases" framework.

@telegraf-tiger telegraf-tiger bot added fix pr to fix corresponding bug plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Oct 6, 2023
@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Oct 10, 2023
@Hipska
Copy link
Contributor

Hipska commented Oct 11, 2023

This PR adds a convert_string_fields option which by default tries to convert string-fields to numeric values succeeding if they e.g. contain "3.14" or "0" but failing on any non-convertible values like "3.1.2" or "foo". In the former case the value is sent to Redis while in the latter (failing) case the field is dropped but sending is continued. This resembles the current behavior as close as possible. When setting the option to false all string fields are dropped and no attempt is made to safe them.

Well, that's a great idea. There are some other outputs that also don't support strings which could benefit from this. A lot of them are just simply dropping any string, regardless of it's value.

@srebhan
Copy link
Member Author

srebhan commented Oct 11, 2023

@Hipska feel free to reuse the code!

@Hipska
Copy link
Contributor

Hipska commented Oct 11, 2023

Maybe I should list these on a new Issue?

And probably many more..

Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the additional logging during field handling. This might reveal some new "issues" to end-users that are rather verbose. However, this handling is the correct change.

@powersj powersj force-pushed the redistimeseries_issue_14058 branch from 2239ffa to 21ee760 Compare October 12, 2023 16:05
@powersj
Copy link
Contributor

powersj commented Oct 12, 2023

I've rebased on master to resolve the conflicts

@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.08 % for linux amd64 (new size: 211.0 MB, nightly size 208.7 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

@powersj powersj merged commit 68eda25 into influxdata:master Oct 12, 2023
@github-actions github-actions bot added this to the v1.28.3 milestone Oct 12, 2023
powersj pushed a commit that referenced this pull request Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix pr to fix corresponding bug plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Redis Timeseries' TS.ADD only accepts numeric values.
3 participants