Fix issue for HttpClient connection closed unexpectedly error after events submission.

[edwardmeng]

We have encountered errors as following

2019-03-15 22:26:45.6781 Info  DefaultEventQueue: Sent 27 events to "https://log.mysite.com/".
2019-03-15 22:26:55.6419 Error DefaultEventQueue: An error occurred while submitting events.
2019-03-15 22:26:55.6439 Error System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: 基础连接已经关闭: 服务器关闭了本应保持活动状态的连接。 ---> System.IO.IOException: 无法从传输连接中读取数据: 远程主机强迫关闭了一个现有的连接。。 ---> System.Net.Sockets.SocketException: 远程主机强迫关闭了一个现有的连接。
   在 System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   在 System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- 内部异常堆栈跟踪的结尾 ---
   在 System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)
   在 System.Net.TlsStream.EndRead(IAsyncResult asyncResult)
   在 System.Net.PooledStream.EndRead(IAsyncResult asyncResult)
   在 System.Net.Connection.ReadCallback(IAsyncResult asyncResult)
   --- 内部异常堆栈跟踪的结尾 ---
   在 System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   在 System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- 内部异常堆栈跟踪的结尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Exceptionless.Submission.DefaultSubmissionClient.PostEvents(IEnumerable`1 events, ExceptionlessConfiguration config, IJsonSerializer serializer)
2019-03-15 22:26:55.6449 Info  DefaultEventQueue: Suspending processing for: 00:05:00.

The root cause is the http client requests will send header Connection:Keep-Alive.

[ejsmith]

Hmm... can you please tell me what this does and why we need it? I see that it instructs the HttpClient to close the connection and not use keep alive, but why would we want to need to do that?

[shamork]

We should revert this commit, which can cause port exhausting problem.
Add a retry logic to fix System.Net.Http.HttpRequestException ( when connection is killed or reset unexpectly ).

[niemyjski]

@shamork do you have a link to more information about this? @edwardmeng did quite a bit of testing on this and we haven't had any issues reported since we've made this change and some clients are under extreme load.

[shamork]

Connection: Close means disable http-keep-alive feature
https://www.imperva.com/learn/performance/http-keep-alive/
https://serverfault.com/questions/453576/is-tcp-port-exhaustion-real
https://docs.microsoft.com/en-us/windows/client-management/troubleshoot-tcpip-port-exhaust

[niemyjski]

@shamork Are you currently running into this issue? Can you give us steps and criteria you are seeing this under? We made this change because we wanted more control over how connections were managed, All http submission happens on a background thread on a timer so it's not constant requests being made.

[ejsmith]

@shamork we are on a timer and send in batches so it's not constantly sending events. If I remember correctly, I looked at what MS was doing for App Insights client and they were doing the same thing.