ETCD container mount /etc/hosts file

[catherinetcai]

This PR just volume mounts the /etc/hosts file from the masters into the etcd-server containers. I'm not 100% sure if this is the right approach to fixing this problem, but I was running into the issue of the etcd servers no longer being able to discover each other after performing a rolling-update. I'm running this version of protokube locally and it's fixed my issues of the etcd containers not being able to discover each other after an update.

I saw that the kube-proxy manifest was already volume mounting the /etc/hosts file, so I figured this was kosher.

[k8s-ci-robot]

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://github.com/kubernetes/kubernetes/wiki/CLA-FAQ to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
• If you have done the above and are still having issues with the CLA being reported as unsigned, please email the CNCF helpdesk: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

Hi @catherinetcai. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[chrislovecnm]

I don't think this is a bad idea, but upstream is not doing this. https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/etcd/etcd.manifest

Protokube sets the etcd dns name, what is being mounted for the hosts file?

[chrislovecnm]

Also are you using gossip?

[catherinetcai]

Yes, I am using gossip.

Also, sorry, I realized I didn't provide enough context around the original PR for it to make sense.

To my understanding, when a Docker container is running on the host network, it copies from the host's /etc/hosts file into the container. The /etc/hosts file in the container is editable in the container without impacting the host. The side effect to that is any subsequent changes made in /etc/hosts on the host does not propagate to the container.

I think what's happening is that on rolling updates, protokube is updating the /etc/hosts file, but these updates aren't making it to the etcd-server/etcd-server-events containers that come up. As a result, rolling updates for me started always failing, since the etcd containers couldn't discover each other.

[chrislovecnm]

@justinsb can comment on this better, but I am thinking we only want to do this if we are using gossip.

@gambol99 / @KashifSaadat / @andrewsykim any opinions?

[chrislovecnm]

Talked with @justinsb, he is going to take a look.

[justinsb]

Thank you so much @catherinetcai - you are absolutely right. I was very confused, because I didn't see how an HA cluster would come up at all, but I'm guessing there's actually a race condition that lets us get away with it, but without your PR not all the nodes of an HA cluster reliably come up. Working towards getting that under e2e testing, you'll be please to hear!

/lgtm

/ok-to-test

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• OWNERS [justinsb]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[k8s-github-robot]

/test all [submit-queue is verifying that this PR is safe to merge]

[k8s-github-robot]

Automatic merge from submit-queue. .