Moderate loads cause the Proxy server DNS resolving to fail

[r0wbrt]

See issue golang/go#18588, apparently resolving a moderate number of dns queries causes the resolver to consume all of the file descriptors of the program. To reproduce, open three major news sites via news.google.com at the same time. You will get a list of errors. This is further aggravated by the loop back and localhost connection guards which also do a DNS lookup. So each HTTP request is actually performing at a min 3 dns lookups. Preliminary experimentation where the URL's host name is replaced by the IP and the web server host header is set to the original hostname did not work. The TLS library seems to use the host name in the URL field for validation.

Suggest Fix : Implement a custom resolver that uses a single port (And hence a single OS handle) to do the DNS resolving. Probably could fork the Go resolver for this. Also add some sort of DNS caching into the mix.

2017/12/28 23:19:32 Get https://web.archive.org/web/20140801161459js_/http://cdn.vidible.tv/prod/tags/2014-06/bid=52a8806de4b0020fd7123572/pid=539b08a0e4b0998910590862/51b6b137e4b024cd0d33d90c.js: dial tcp 207.241.225.186:443: getsockopt: connection refused
2017/12/28 23:19:56 lookup z.moatads.com on 127.0.0.53:53: no such host
2017/12/28 23:19:57 lookup www.nbcnews.com on 127.0.0.53:53: no such host
2017/12/28 23:19:57 lookup dt.adsafeprotected.com on 127.0.0.53:53: no such host
2017/12/28 23:19:57 lookup dt.adsafeprotected.com on 127.0.0.53:53: no such host
2017/12/28 23:19:57 lookup d3qdfnco3bamip.cloudfront.net on 127.0.0.53:53: no such host
2017/12/28 23:19:57 Get https://nervoussummer.com/405ff492c0b37410bb6609de1410d2c091f468e4e684464cd8d9f2a1661f0fed65f9a53facf9a476f78a4aaf50a84287bd6b67308ff3cad6abfa5b67dec755: dial tcp: lookup nervoussummer.com on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup sc.iasds01.com on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup sc.iasds01.com on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup logx.optimizely.com on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup secure-us.imrworldwide.com on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup 0914.global.ssl.fastly.net on 127.0.0.53:53: no such host
2017/12/28 23:19:58 lookup cnn.saambaa.com on 127.0.0.53:53: no such host
2017/12/28 23:19:59 lookup secure-dcr.imrworldwide.com on 127.0.0.53:53: no such host
2017/12/28 23:19:59 lookup srv-2017-12-29-05.config.parsely.com on 127.0.0.53:53: no such host
2017/12/28 23:20:00 lookup log.outbrain.com on 127.0.0.53:53: no such host
2017/12/28 23:20:00 lookup w.usabilla.com on 127.0.0.53:53: no such host
2017/12/28 23:20:00 lookup px.moatads.com on 127.0.0.53:53: no such host
2017/12/28 23:20:00 lookup px.moatads.com on 127.0.0.53:53: no such host
2017/12/28 23:20:01 lookup nvcdn.nbcnews.com on 127.0.0.53:53: no such host
2017/12/28 23:20:01 lookup www.washingtonpost.com on 127.0.0.53:53: no such host
2017/12/28 23:20:01 lookup aa.agkn.com on 127.0.0.53:53: no such host
2017/12/28 23:20:02 http: TLS handshake error from 127.0.0.1:39524: tls: first record does not look like a TLS handshake
2017/12/28 23:20:02 lookup b.scorecardresearch.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup geo.moatads.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup ad.doubleclick.net on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup ml314.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup ib.adnxs.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup match.adsrvr.org on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup p.adsymptotic.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup bea4.cnn.com on 127.0.0.53:53: no such host
2017/12/28 23:20:03 lookup su.addthis.com on 127.0.0.53:53: no such host
2017/12/28 23:20:04 lookup cnn.bounceexchange.com on 127.0.0.53:53: no such host
2017/12/28 23:20:04 lookup soda.startappservice.com on 127.0.0.53:53: no such host
2017/12/28 23:20:04 lookup b.scorecardresearch.com on 127.0.0.53:53: no such host
2017/12/28 23:20:04 lookup global.ib-ibi.com on 127.0.0.53:53: no such host
2017/12/28 23:20:04 lookup qsearch.media.net on 127.0.0.53:53: no such host

[r0wbrt]

The back off algorithm has really helped to solve this problem when the server is under light usage by a single client. Before, these errors would occur under light usage from a single client.
c0907d4