Use the maximum jitter when calculating the timeout

[jameshartig]

Description

This is a follow-up to #11500 and fixes #13695. @kisunji actually added back the RandomStagger in c8897b2

The timeout should include the maximum possible jitter since the server will randomly add to it's timeout a jitter. If the server's timeout is less than the client's timeout then the client will return an i/o deadline reached error.

Testing & Reproduction steps

Before:

time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
rpc error making call: i/o deadline reached
real    10m11.469s
user    0m0.018s
sys     0m0.023s

After:

time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
[...]
real    10m35.835s
user    0m0.021s
sys     0m0.021s

PR Checklist

• updated test coverage
• external facing docs updated
• not a security concern

[hashicorp-cla]

All committers have signed the CLA.

[kisunji]

Thank you for debugging this issue! This certainly looks like the culprit of the timeout issues. Will merge when tests are green

[kisunji]

We have some CI integration tests that don't seem to work for forked repos; I'll run the tests on a local branch and merge.

[jkirschner-hashicorp]

This fix will be available in:

• 1.12 release series: 1.12.5+
• 1.13 release series: 1.13.2+
• 1.14.0+

[jofowler]

I own the code for a Go client that uses the Hashicorp consul Go api. We are currently encountering this error a lot. I see that a fix will be out soon. Is there a change that I could make on the client side to avoid this error? If so, which timeout value would I need to change and what should I change it to? We are using the Consul 1.12.2 server and the 1.12.0 consul client api. Thanks.