increase docker-healthcheck respose timeout

[tatobi]

In case of increased I/O load, the 10sec timeout is not enough on small / heavily loaded systems thus I propose the 60sec. The kubelet timeout is 2m (120sec) by default to detect health problems. Secondly, the docker restart can load heavily the host OS even huge systems because of many pods initialization at the same time. Continuous dockerd restart loop - a deadlock of node - is observed. Thirdly, because of the forcibly closed sockets and the kernel TCP TIME_WAIT value, the TCP sockets are not usable immediately with a "restart", wait for FIN_TIMEOUT is necessary before start services.
Workaround #1 for: #5434

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[ihoegen]

Also @tatobi be sure to sign the CLA.

/ok-to-test

[ihoegen]

/lgtm

[chrisz100]

Can we make this timeout dynamic instead? I can see people who don’t want to wait 60 seconds here and prefer getting a failed early.

[tatobi]

We could, but we should find precisely the metrics which affect the dynamism. The current operation is a bit rough of course, we could check the running container's statuses instead - maybe in next version. Do you think monitoring all pod's status after a stop / start operation would be more appropriate?

[chrisz100]

I wouldn't say we need to get down to that level as that'd be quite compute costly. But making the timeout dynamic could allow to make this specific to certain environments and their specifics.

[justinsb]

/retest

I think we should get this in, I think the behaviour here is much better - fewer false positives and likely much more visible because of the /proc/sys/net/ipv4/tcp_fin_timeout sleep.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ihoegen, justinsb, tatobi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

The errors travis is picking up are the old mis-spellings, because travis doesn't test the code after rebasing.

Force-merging.