Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

influx format rejects newlines in quoted field values #4942

Closed
awelzel opened this issue Oct 31, 2018 · 5 comments · Fixed by #5484
Closed

influx format rejects newlines in quoted field values #4942

awelzel opened this issue Oct 31, 2018 · 5 comments · Fixed by #5484
Assignees
Labels
area/influxdb bug unexpected problem or unintended behavior
Milestone

Comments

@awelzel
Copy link

awelzel commented Oct 31, 2018

Hey hey,

I am trying to send log messages through telegraf via UDP in the influx line format. It appears the telegraf influx data parser does not accept newlines inside field values - not even inside quoted field values. I believe this should be allowed. In fact, using the same steps to reproduce the issues directly with an influxdb udp listener setup works as expected - newlines are stored in the database.

Relevant telegraf.conf:

[[inputs.socket_listener]]
  service_address = "udp://:9089"
  data_format = "influx"

influxdb.conf for comparison

[[udp]]
  enabled = true
  bind-address = ":8089"

System info:

Telegraf: master (d2717f7)
Debian buster/sid

Steps to reproduce:

    # Sending to telegraf. Note the -e
    $ echo -e 'test_logs message="Hey!\n"' | nc -u localhost 9089

    # telegraf log:
    2018-10-31T12:10:10Z E! Error in plugin [inputs.socket_listener]: unable to parse incoming packet: metric parse error: expected field at offset 23: "test_logs message=\"Hey!\n\"\n"

Sending to influxdb directly works as expected:

    $ echo -e 'test_logs message="Hey!\nhow are you?"' | nc -u localhost 8089
    > select * from test_logs;
    name: test_logs
    time                message
    ----                -------
    1540988837240091217 Hey!
    how are you?

Expected behavior:

The influx parser should accept newlines inside quoted field values.

Actual behavior:

The parser rejects newlines inside quoted field values.

Additional info:

From what I could gather from the code: Telgraf simply scans for a newline character in the received data to determine where a influx line starts and terminates. The influxdb code seems more advanced in that it handles quoted fields: https://github.com/influxdata/influxdb/blob/master/models/points.go#L1081

Background:

I'm attempting to send log messages (and particularly tracebacks with newlines) using a Python logging handler [1]. Previously I had only tested against influxdb directly, but in the meantime have a use case to send logs through telegraf which should then get forwarded to influxdb.

[1] https://github.com/awelzel/influxpy

@glinton glinton added area/influxdb bug unexpected problem or unintended behavior labels Oct 31, 2018
@danielnelson
Copy link
Contributor

As a workaround string fields support the \n escape to insert a CR.

@awelzel
Copy link
Author

awelzel commented Oct 31, 2018

As a workaround string fields support the \n escape to insert a CR.

Daniel, thanks for the quick reply! Hmm, I have just tried sending a \n (note the missing -e for echo).

$ echo 'test_logs message="Hey!\nhow are you?"' | nc -u localhost 9089

It still causes a parse error:

2018-10-31T19:31:01Z E! Error in plugin [inputs.socket_listener]: unable to parse incoming packet: metric parse error: expected field at offset 24: "test_logs message=\"Hey!\\nhow are you?\"\n"

Further note, sending \n to an influxdb listener causes \n to show up in the field value - there is no CR. Seems influxdb does not support the \n escape inside field values.

Hmm... thoughts?

@danielnelson
Copy link
Contributor

It looks like I was wrong, the only currently supported escapes for strings fields in our parser are \\ and \".

@awelzel
Copy link
Author

awelzel commented Nov 1, 2018

It looks like I was wrong, the only currently supported escapes for strings fields in our parser are \\ and \".

Ok, thanks - I gather there's no possibility to send CRs inside field values at the moment..

@danielnelson
Copy link
Contributor

No sorry, it's not possible currently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/influxdb bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants