inputs.http does not set Content-Length header correctly

[marlinkdj]

Relevant telegraf.conf

[[inputs.http]]
  
  ## List of urls to query.
  urls = ["URL_WAS_HERE"]
  
  ## HTTP Request Method
  method = "POST"
  
  ## Optional HTTP headers
  headers = {"Content-Length" = "50"}	
  
  ## Optional HTTP Request Body
  body = '''{"F_":"LOGIN","DATA":{"ID":"guest","PWD":"guest"}}'''
  
  ## Optional TLS Config
  insecure_skip_verify = true

Logs from Telegraf

root@servername:~# telegraf --config /etc/telegraf/telegraf.d/http_input_login_V85NX_test.conf  --test --debug
2022-04-26T11:09:35Z I! Starting Telegraf 1.22.2
2022-04-26T11:09:35Z I! Loaded inputs: http
2022-04-26T11:09:35Z I! Loaded aggregators:
2022-04-26T11:09:35Z I! Loaded processors:
2022-04-26T11:09:35Z W! Outputs are not used in testing mode!
2022-04-26T11:09:35Z I! Tags enabled: host=servername
2022-04-26T11:09:35Z D! [agent] Initializing plugins
2022-04-26T11:09:35Z D! [agent] Starting service inputs
2022-04-26T11:09:38Z E! [inputs.http] Error in plugin: [url=URL_WAS_HERE]: received status code 411 (Length Required), expected any value out of [200]
2022-04-26T11:09:38Z D! [agent] Stopping service inputs
2022-04-26T11:09:38Z D! [agent] Input channel closed
2022-04-26T11:09:38Z D! [agent] Stopped Successfully
2022-04-26T11:09:38Z E! [telegraf] Error running agent: input plugins recorded 1 errors

System info

Telegraf: 1.22.2 , Operating System: Debian GNU/Linux 11 (bullseye), Kernel: Linux 5.10.0-8-amd64

Docker

No response

Steps to reproduce

1. Tried the above config with no optional headers - Error code 411
2. Adding a optional HTTP header with Content-length does not work either - Error code 411
3. Tried using body=1 and Content-length =1, still same result. - Error code 411
4. Tried body=1 and Content-length =1 with cURL and get error code 500 (As expected).

Expected behavior

Reply from API with JSON containing login details.

{
        "RESULT":     1,
        "SID":        19463950,
        "LEVEL":      "GUEST"
}

Actual behavior

Reply is:

Error in plugin: [url=URL_WAS_HERE]: received status code 411 (Length Required), expected any value out of [200]

Additional info

Trying to pull data from an API that requires the Content-Length header in the POST request.

Tried with similiar .conf file using the HTTP_Response inputs plugin and this works as intended towards the same API endpoint.

.conf file:

[[inputs.http_response]]
  ## List of urls to query.
  urls = ["URL_WAS_HERE"]

  ## HTTP Request Method
  method = "POST"

  ## Optional HTTP Request Body
  body = '{"F_":"LOGIN","DATA":{"ID":"guest","PWD":"guest"}}'

  response_body_field = 'Test'
  
  insecure_skip_verify = true

Response:

root@servername:~# telegraf --config /etc/telegraf/telegraf.d/http_response_login_V85NX_test.conf  --test --debug
2022-04-26T11:15:02Z I! Starting Telegraf 1.22.2
2022-04-26T11:15:02Z I! Loaded inputs: http_response
2022-04-26T11:15:02Z I! Loaded aggregators:
2022-04-26T11:15:02Z I! Loaded processors:
2022-04-26T11:15:02Z W! Outputs are not used in testing mode!
2022-04-26T11:15:02Z I! Tags enabled: host=servername
2022-04-26T11:15:02Z D! [agent] Initializing plugins
2022-04-26T11:15:02Z D! [agent] Starting service inputs
2022-04-26T11:15:05Z D! [agent] Stopping service inputs
> http_response,host=servername,method=POST,result=success,server=URL_WAS_HERE,status_code=200 Test="{
        \"RESULT\":     1,
        \"SID\":        19463950,
        \"LEVEL\":      \"GUEST\"
}",content_length=53i,http_response_code=200i,response_time=3.100586043,result_code=0i,result_type="success" 1650971706000000000

[jhychan]

Given that the HTTP client is just the standard golang net/http module, pretty unlikely that it's broken and handling payload content length incorrectly. Have tested against other HTTP endpoints and HTTP POSTs work just fine.

Took a look in wireshark of http vs http_response and I see that http plugin POSTs data with Transfer-Encoding: chunked, while http_response plugin does not. HTTP payload is formatted correctly for both methods. The reason why is unclear but not unsurprising, as HTTP client config is slightly different between the two plugins.

So I'm betting the root cause of your issue here is the API server - it has an incomplete HTTP 1.1 implementation and it doesn't know how to handle chunked requests.

[marlinkdj]

Thanks for your reply.

Can I change the Transfer-Encoding used by the http plugin?
ie:

Optional HTTP headers

headers = {"Transfer-Encoding" = "gzip"}

?

[jhychan]

Doesn't look like there's a way to change this behaviour in Telegraf config. But I think I've found the cause:
golang/go#34295

I think this NopCloser on the io.reader is causing the http Request to not be able to determine the body's content length:
https://github.com/influxdata/telegraf/blob/v1.22.2/plugins/inputs/http/http.go#L267

Which in turn causes the http Request client to use chunked transfer encoding.

In contast to http_response, which doesn't do this:
https://github.com/influxdata/telegraf/blob/v1.22.2/plugins/inputs/http_response/http_response.go#L268

Don't know anything about io readers/writers and closers, so I'll leave it to Telegraf devs from here.

[powersj]

Can I change the Transfer-Encoding used by the http plugin?

Since Telegraf is not explicitly setting this value, can you try this and see what happens?

[marlinkdj]

Hi @powersj.
Results after adding the headers = {"Transfer-Encoding" = "compress\gzip\deflate"} option to above .conf file is still the same 411 error from the endpoint.

However the reason for this could still be that the endpoint only accept\assumes "chucked" Encoding with a valid Content-length header attached.

[powersj]

Thanks for trying that.

As @jhychan determined, since we are using a ReadCloser the HTTP request is unable to determine the exact content length.

If body is of type *bytes.Buffer, *bytes.Reader, or *strings.Reader, the returned request's ContentLength is set to its exact value

I believe the use of the NopCloser is due to the use of the optional gzip content-encoding. The CompressWithGzip function returns a ReadCloser. If we wanted to ensure the body was in one of the three types we would have to convert the ReadCloser into one of those three. I do not have much experience with passing between readers like this or the potential effects/unintended consequences of doing so.

@reimda thoughts?

[jhychan]

Hi @powersj. Results after adding the headers = {"Transfer-Encoding" = "compress\gzip\deflate"} option to above .conf file is still the same 411 error from the endpoint.

However the reason for this could still be that the endpoint only accept\assumes "chucked" Encoding with a valid Content-length header attached.

HTTP spec forbids sending Content-Length if using chunked transfer encoding. They are mutually exclusive - it should be one or the other, never both. A client/server that sends both would be sending an invalid request/response - the server/client would not know how large the payload actually is. So the http module is probably going to replace/strip out those headers it has to manage.

Can I change the Transfer-Encoding used by the http plugin?

Since Telegraf is not explicitly setting this value, can you try this and see what happens?

I'd expect Transfer-Encoding to be overridden or stripped out by the http module. The HTTP body contains chunk lengths when using chunked transfers.

Path forward:

1. Fix API endpoint to support chunked requests
2. Change inputs.http plugin to avoid chunking. Doubt there's any real use case that would require this. The body content is read into memory when the TOML configuration is loaded anyway, so it's kind of pointless I think? The reason to use chunked encoding is for very large buffers (ie. files), where you wouldn't want to load the entire file into memory just to transfer it.

[marlinkdj]

Sorry if I'm dumb here, but my understanding and drilldown of the issue is as follow:

My API only accept POST requests with Content-Length header.

inputs.http_response works fine and sends the expected headers with the correct "Content-length" of the body in the POST request.

inputs.http is unable to calculate the correct Content-length and the POST request is sent without the Content-length header.(Probably chunked?)

Summarize:

inputs.http plugin does NOT support Content-length headers.

Please feel free to comment or correct me as I'm by no means any expert, but here to learn.

[marlinkdj]

Did some more testing:
I set up a API endpoint for testing and catching the headers.

It's possible to set your custom headers, but if you add "Content-Length" the inputs.http plugin strips it.

[marlinkdj]

.conf file:

[[inputs.http]]	
  ## List of urls to query.
  urls = ["URLWASHERE/debug"]

  ## HTTP Request Method
  method = "POST"
  
  ## HTTP Content-Encoding for write request body, can be set to "gzip" to
  ## compress body or "identity" to apply no encoding
  content_encoding = "gzip"
  
  headers = { "Content-Length" =  "50" }
  headers = { "Content-Type" =  "application/json" }
  headers = { "This_is_a_testheader" = "IT_IS_VISIBLE"}
 

Result at API:

User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Content-Encoding: gzip
Content-Type: application/json
This-Is-A-Testheader: IT_IS_VISIBLE
Accept-Encoding: gzip

[powersj]

@Badegakken, what is the target you are trying to hit that requires a content-length?

I have put up #11083 that switches from ReadCloser to a Reader can you try that out once artifacts are attached to the PR? That should happen in 20-30mins from this post once tests are run. A comment will get added by the Telegraf Tiger with links to artifacts you can download and then try.

Thanks again!

[marlinkdj]

@powersj , Thanks a lot, I will try it later today.

Endpoint is a third party API running on a VSAT antenna controller.