Limit maximum response time

[krizhanovsky]

Relates to #712.

Limit maximum response time

Currently Tempesta can wait infinitly for a backend response skewing APM statistics. There must be a configuration option for maximum time in seconds to wait for a backend response. If Tempesta does not receive any data from the external application within this timeout limit, it will mark this connection as bad. Nginx does this with proxy_read_timeout.

LB: exponencial retry timeout backoff

This issue requires a timer, which also can be used to postpone a request retry on failed backend connection (#1136). Currently we retry requests immediately, but it can lead to overloading of already overloaded servers. So we should retry, within the defined timeout, in exponential backoff manner.

Testing

Please also create a functional test or make an issue in https://github.com/tempesta-tech/tempesta-test/

[krizhanovsky]

There are at least 2 cases for the upstream timeouts:

1. we can't send a response, probably can be handled with sysctl on TCP keepalive
2. TCP is fine, but the upstream is stuck on response processing - this seems should be fixed separately

Maybe not for good to start