Feature/http response configurable proxy #3374

[winem]

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

These 2 commits fix #3374

[danielnelson]

Looks good but I wonder if we need support for no_proxy too.

@cpoule23 can you provide a bit more information about your setup and how you will use this?

[cpoule23]

Hello Danielnelson,

I'm using http_response inside a company with proxy usage for security/flexibility reasons.

I need to test some internal URL so I have to reach them without proxy and internet URL so with proxy. Without "NO_PROXY" settings, all my requests (internal too) will try to reach firstly the proxy and he will not find the URL as he will try to go on internet too find them.

NO_PROXY for my internal domain (intranet, home made application), proxy for all others on public URL.

I hope it's clear for the need ?

[winem]

@cpoule23 so you want to be able to override the system wide proxy by "no proxy" for internal domains, right? I think that's indeed a use-case for no_proxy.

[cpoule23]

yes :)
I try to add environnment variable for no_proxy/http_proxy in ~./bashrc but it doesn't work for http_response :(

Or maybe it's not a good way to proceed

[danielnelson]

@cpoule23 You can add these environment variables to /etc/default/telegraf to have them picked up by the service, but you will have to use the same values for all of Telegraf.

[danielnelson]

@cpoule23 Just to be clear, do you need a per plugin no_proxy setting or is a global no_proxy setting across all Telegraf plugins good enough?

[cpoule23]

Hello,
I already add http_proxy + no_proxy in /etc/default/telegraf but it's not working for public URLs (timeout), only OK internal one's.

If you add http & https_proxy settings inside http_response and if internal will use the /etc/default/telegraf for no_proxy setting it's fine for me :)

[danielnelson]

You won't be able to mix the settings location with this patch. For instance, you won't be able to use the no_proxy setting in /etc/default/telegraf along with a http_proxy setting from the config.

It sounds like perhaps you don't need this patch, but instead have an issue with the environment variables not working as expected. Can you let me know what your env vars are set to and one of the urls you are having trouble with, describe what you expect to happen and what telegraf actually does.

[cpoule23]

Hello,

• On /etc/default/telegraf I have :

http_proxy=http://myproxyURL:3128
https_proxy=http://myproxyURL:3128
no_proxy='localhost','10.*','172.16.*.*','192.168.*.*','127.0.0.1'

• telnet on myproxyURL 3128 is working fine

• when I set proxy setting, I can reach http://www.lemonde.fr by CLI

• I add in telegraf this configuration (telegraf.d):

# HTTP/HTTPS request given an address a method and a timeout
[[inputs.http_response]]
  address = "http://www.lemonde.fr"
  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"
  ## HTTP Request Method
  # method = "GET"
  ## Whether to follow redirects from the server (defaults to false)
  # follow_redirects = false
  [inputs.http_response.tags]
    application = "LEMONDE"
    server_name="lemonde"

• And on my influx, I get a timeout :/
  

• I have no more logs in telegraf.log to understand what's going wrong but I saw no input inside my squid logs.

[danielnelson]

I think you shouldn't have the single quotes in no_proxy, though I'm not sure that this would make it so the proxy is not working for proxy sites:

no_proxy="localhost,10.*,172.16.*.*,192.168.*.*,127.0.0.1"

Can you run these commands from bash:

unset http_proxy
unset https_proxy
unset no_proxy
http_proxy=http://myproxyURL:3128 curl http://www.lemonde.fr -v -o /dev/null
http_proxy=http://myproxyURL:3128 telegraf --input-filter http_response --test

[danielnelson]

Let's change this to http_proxy to match the variable name in other plugins.

[winem]

Ok, sounds legit. Did that and will push it right after I tested it.

[cpoule23]

Hello Danielnelson,

-/etc/default/telegraf

[root@myserver kapacitor]# cat /etc/default/telegraf
http_proxy=http://myproxy:3128
https_proxy=http://myproxy:3128
no_proxy=localhost

-telegraf.conf

[[inputs.system]]
# HTTP/HTTPS request given an address a method and a timeout
[[inputs.http_response]]
  address = "http://www.lemonde.fr"

  ## Set response_timeout (default 5 seconds)
  # response_timeout = "5s"

  ## HTTP Request Method
  # method = "GET"

  ## Whether to follow redirects from the server (defaults to false)
  # follow_redirects = false

  [inputs.http_response.tags]
    application = "lemonde"
    server_name="front"

-http_proxy=http://myproxyURL:3128 curl http://www.lemonde.fr -v -o /dev/null

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to proxy myproxy port 3128 (#0)
*   Trying [myIP]...
* Connected to myproxy (myIP) port 3128 (#0)
> GET http://www.lemonde.fr/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.lemonde.fr
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 302 Found
< Server: squid/3.5.20
< Date: Mon, 15 Jan 2018 09:12:43 GMT
< Content-Length: 0
< Location: http://proxy-redirect.mydomain/cgi-bin/URLblocked.cgi?color=white&size=normal&clientaddr=10.223.24.42&clientname=10.223.6.208&clientuser=unknown&clientgroup=allSystems&category=news&url=http%3A%2F%2Fwww.lemonde.fr%2F
< X-Cache: MISS from myproxy
< X-Cache-Lookup: MISS from myproxy:3128
< Via: 1.1 myproxy (squid/3.5.20)
< Connection: keep-alive
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host myproxy left intact

-http_proxy=http://myproxy:3128 telegraf --input-filter http_response --test

2018/01/15 10:17:28 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.http_response, Collection 1
> http_response,server=http://www.lemonde.fr,method=GET,application=lemonde,server_name=front,host=myserver result_type="timeout" 1516007853000000000

I have a timeout to reach http://www.lemonde.fr :/
On my squid logs, I can see the curl access but not the access through telegraf command (http_proxy=http://myproxy:3128 telegraf --input-filter http_response --test)

Do you have an idea please ?

[danielnelson]

@cpoule23 I'm not sure why it is not working, it looks like a bug so I opened #3690. I'll have to try to duplicate this issue or provide a way to get more debugging info.

[winem]

Pushed the change to http_proxy to be consistent with existing plugins supporting this setting.

[winem]

@danielnelson I tested and pushed the latest code. How to proceed regarding the CircleCI failures? As far as I see it, we don't have much influence on the failures because they're caused by other modules, etc. Is there anything else from my / our side required before this pull request can be accepted?

[danielnelson]

The last failure in particular can be fixed by rebasing against master, but sometimes we have intermittent failure due to unfixed bugs in tests, in the future don't worry about these too much as I can rerun the builds manually if I see them.

[hkadakia]

What if the http proxy has username and password? Is there a way we can add those as part of the config.

[danielnelson]

@hkadakia Try setting the username and password in the userinfo section of the URL:

export http_proxy=http://myusername:[email protected]

You may need to escape special characters in the password, if you need more help please ask on the InfluxData Community site.

[winem]

@hkadakia, I'm sorry for the late reply. I missed the notification. The way described by @danielnelson was tested and should work.