Bind the kubelet to the local ipv4 address #4417

dezmodue · 2018-02-09T23:28:19Z

No description provided.

chrislovecnm · 2018-02-12T21:07:36Z

@liwenwu-amazon double checking with you that this was the recommended solution.

/ok-to-test

bksteiny · 2018-02-26T01:09:31Z

@chrislovecnm, I tested this based on our Slack conversation , and --node-ip is added to the kubelet environment file.

root@ip-10-4-233-250:/home/admin# cat /etc/sysconfig/kubelet 
DAEMON_ARGS="--allow-privileged=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=101.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true --hostname-override=ip-10-4-233-250.us-west-2.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kubernetes.io/role=master,node-role.kubernetes.io/master= --non-masquerade-cidr=101.64.0.0/10 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --register-with-taints=node-role.kubernetes.io/master=:NoSchedule --require-kubeconfig=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/ --node-ip=10.4.233.250"
HOME="/root"

Log from journalctl:

Feb 26 01:04:27 ip-10-4-233-250 kubelet[1663]: I0226 01:04:27.254687    1663 kubelet_node_status.go:455] Using node IP: "10.4.233.250"

However, there is an issue pulling down the amazon-k8s-cni image from ECR. In order to pull it down, I had to:

Add AmazonEC2ContainerRegistryReadOnly permission to my Kops user
Run aws configure to setup my Kops user
Authenticate to ECR using: aws ecr get-login and run docker login ..... https://602401143452.dkr.ecr.us-west-2.amazonaws.com

Is this expected or am I doing something wrong?

chrislovecnm · 2018-02-27T05:40:57Z

nodeup/pkg/model/kubelet.go

+			if err != nil {
+				glog.Fatalf("Couldn't fetch the local-ipv4 address from the ec2 meta-data: %v", err)
+			} else {
+				flags += " --node-ip=" + localIpv4


Don’t need the else as I think glog.Fatal will return an err. We may want to log and error and return err here.

chrislovecnm · 2018-02-27T05:42:11Z

@bksteiny no idea on the image, I would reach out to the AWS folks

dezmodue · 2018-02-27T08:42:40Z

@chrislovecnm changed as requested

chrislovecnm · 2018-02-27T20:10:29Z

Run ./hack/update-bazel.sh

You need to rebase and run that command as CI is failing.

chrislovecnm · 2018-02-27T20:10:55Z

Looks good ... CI is not happy

justinsb · 2018-02-28T05:58:50Z

Code looks fine once CI is happy.

Another option is to do this (though with local-ipv4), as it is just an HTTP request I believe: https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/nodeup/command.go#L344-L348

dezmodue · 2018-02-28T13:29:56Z

@justinsb let me know if I should update the PR as you suggested, I don't have a preference

chrislovecnm · 2018-02-28T16:45:29Z

Let’s just get CI fixed ;)

chrislovecnm · 2018-02-28T16:46:16Z

You need to run the command to fix bazel. I mentioned it in a previous comment.

chrislovecnm · 2018-02-28T16:47:20Z

@justinsb this needs to go into 1.9 since it is fixing a bug with the aws cni provider

liwenwu-amazon · 2018-02-28T19:28:55Z

I pulled this PR locally and build a dev version of kop. I am NOT able to bring up a kop cluster successfully. Not sure it is my environment issue or this PR.
Here are output

ubuntu@ip-10-0-1-11:~/workspace/src/k8s.io/kops$ kops update cluster cni-feb28.k8s-test.com --yes
W0228 19:10:04.388084   25171 apply_cluster.go:778] unable to parse kops version "dev"
W0228 19:10:04.404572   25171 urls.go:71] Using base url from KOPS_BASE_URL env var: "https://k8s-test-com-state-store.s3.amazonaws.com/kops/dev/"
I0228 19:10:04.460920   25171 dns.go:92] Private DNS: skipping DNS validation
I0228 19:10:04.612321   25171 executor.go:91] Tasks: 0 done / 77 total; 30 can run
I0228 19:10:04.880869   25171 vfs_castore.go:715] Issuing new certificate: "ca"
I0228 19:10:05.062297   25171 vfs_castore.go:715] Issuing new certificate: "apiserver-aggregator-ca"
I0228 19:10:05.707267   25171 executor.go:91] Tasks: 30 done / 77 total; 24 can run
I0228 19:10:06.197135   25171 vfs_castore.go:715] Issuing new certificate: "kube-controller-manager"
I0228 19:10:06.432073   25171 vfs_castore.go:715] Issuing new certificate: "kubecfg"
I0228 19:10:06.680989   25171 vfs_castore.go:715] Issuing new certificate: "kube-proxy"
I0228 19:10:06.698087   25171 vfs_castore.go:715] Issuing new certificate: "kubelet-api"
I0228 19:10:06.783840   25171 vfs_castore.go:715] Issuing new certificate: "kops"
I0228 19:10:06.854512   25171 vfs_castore.go:715] Issuing new certificate: "master"
I0228 19:10:06.880286   25171 vfs_castore.go:715] Issuing new certificate: "apiserver-proxy-client"
I0228 19:10:06.932544   25171 vfs_castore.go:715] Issuing new certificate: "kube-scheduler"
I0228 19:10:07.045471   25171 vfs_castore.go:715] Issuing new certificate: "kubelet"
I0228 19:10:07.128042   25171 vfs_castore.go:715] Issuing new certificate: "apiserver-aggregator"
I0228 19:10:07.523679   25171 executor.go:91] Tasks: 54 done / 77 total; 21 can run
I0228 19:10:07.695420   25171 launchconfiguration.go:333] waiting for IAM instance profile "nodes.cni-feb28.k8s-test.com" to be ready
I0228 19:10:07.747283   25171 launchconfiguration.go:333] waiting for IAM instance profile "masters.cni-feb28.k8s-test.com" to be ready
I0228 19:10:18.184210   25171 executor.go:91] Tasks: 75 done / 77 total; 2 can run
I0228 19:10:18.901682   25171 executor.go:91] Tasks: 77 done / 77 total; 0 can run
I0228 19:10:18.901717   25171 dns.go:153] Pre-creating DNS records
I0228 19:10:19.156283   25171 update_cluster.go:253] Exporting kubecfg for cluster
W0228 19:10:19.256569   25171 create_kubecfg.go:58] Did not find API endpoint for gossip hostname; may not be able to reach cluster
kops has set your kubectl context to cni-feb28.k8s-test.com

Cluster is starting.  It should be ready in a few minutes.

here is the output of kops validate error

kops validate cluster
Using cluster from kubectl context: cni-feb28.k8s-test.com

Validating cluster cni-feb28.k8s-test.com

Validation Failed

The dns-controller Kubernetes deployment has not updated the Kubernetes cluster's API DNS entry to the correct IP address.  The API DNS IP address is the placeholder address that kops creates: 203.0.113.123.  Please wait about 5-10 minutes for a master to start, dns-controller to launch, and DNS to propagate.  The protokube container and dns-controller deployment logs may contain more diagnostic information.  Etcd and the API DNS entries must be updated for a kops Kubernetes cluster to start.


Cannot reach cluster's API server: unable to Validate Cluster: cni-feb28.k8s-test.com

The master instance is up already for over 10 minutes.
Here is command I use

kops create cluster --zones us-east-1a,us-east-1b,us-east-1c --dns private --vpc vpc-0066bd79 --node-count 3 --master-size m3.xlarge  --networking amazon-vpc-routed-eni --kubernetes-version 1.9.3 $NAME -v 10

chrislovecnm · 2018-02-28T19:34:06Z

@dezmodue can we get really detailed instructions on how you tested?

…nVPC - #4218

dezmodue · 2018-03-01T19:28:47Z

Hi, at the time I sent in the PR I had built kops and nodeup from the modified version in my repo:

make crossbuild
make crossbuild-nodeup
shasum $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup | cut -d\  -f1 > $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup.sha1
aws s3 cp --acl public-read $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup s3://mybucket-kops-binaries/testing/
aws s3 cp --acl public-read $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup.sha1 s3://mybucket-dev-kops-binaries/testing/
export NODEUP_URL=https://mybucket-kops-binaries.s3.amazonaws.com/testing/nodeup
export NODEUP_HASH=$(cat .build/dist/linux/amd64/nodeup.sha1)

Then I built 2 clusters from a yaml definition like the one in the issue #4218 -- they are still running fwiw

$GOPATH/src/k8s.io/kops/.build/dist/darwin/amd64/kops create -f ${NAME}.yaml

Today I have rebased on 034bad8 and ran again a test cluster:

go version go1.9.2 darwin/amd64
make crossbuild
make crossbuild-nodeup
shasum $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup | cut -d\  -f1 > $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup.sha1
aws s3 cp --acl public-read $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup s3://${BUCKET}-kops-binaries/testing/
aws s3 cp --acl public-read $GOPATH/src/k8s.io/kops/.build/dist/linux/amd64/nodeup.sha1 s3://${BUCKET}-kops-binaries/testing/
export NODEUP_URL=https://${BUCKET}-kops-binaries.s3.amazonaws.com/testing/nodeup
export NODEUP_HASH=$(cat .build/dist/linux/amd64/nodeup.sha1)

export NODE_SIZE=${NODE_SIZE:-m4.large}
export MASTER_SIZE=${MASTER_SIZE:-m4.large}
export ZONES=${ZONES:-"eu-west-1a,eu-west-1b,eu-west-1c"}
$GOPATH/src/k8s.io/kops/.build/dist/darwin/amd64/kops create cluster test-cni.${DOMAIN} \
  --node-count 1 \
  --zones $ZONES \
  --node-size $NODE_SIZE \
  --master-size $MASTER_SIZE \
  --master-zones $ZONES \
  --networking amazon-vpc-routed-eni \
  --kubernetes-version 1.9.3 \
  --topology private \
  --ssh-public-key ~/.ssh/${SSHKEY} \
  --ssh-access ${ACCESS} \
  --api-loadbalancer-type public \
  --authorization rbac \
  --admin-access ${ACCESS} \
  --bastion="true"

The result is a healthy cluster as far as I can tell:

$GOPATH/src/k8s.io/kops/.build/dist/darwin/amd64/kops validate cluster --name test-cni.${DOMAIN}
Validating cluster test-cni.my.domain.com

INSTANCE GROUPS
NAME                    ROLE    MACHINETYPE     MIN     MAX     SUBNETS
bastions                Bastion t2.micro        1       1       utility-eu-west-1a,utility-eu-west-1b,utility-eu-west-1c
master-eu-west-1a       Master  m4.large        1       1       eu-west-1a
master-eu-west-1b       Master  m4.large        1       1       eu-west-1b
master-eu-west-1c       Master  m4.large        1       1       eu-west-1c
nodes                   Node    m4.large        1       1       eu-west-1a,eu-west-1b,eu-west-1c

NODE STATUS
NAME                                            ROLE    READY
ip-172-20-122-136.eu-west-1.compute.internal    master  True
ip-172-20-41-87.eu-west-1.compute.internal      master  True
ip-172-20-87-136.eu-west-1.compute.internal     node    True
ip-172-20-87-29.eu-west-1.compute.internal      master  True

Your cluster test-cni.my.domain.com is ready

I logged into the node and checked the running kubelet process:

admin@ip-172-20-87-136:~$ ps -efw | grep kubelet
root      2503     1  1 15:17 ?        00:00:20 /usr/local/bin/kubelet --allow-privileged=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=172.20.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true --hostname-override=ip-172-20-87-136.eu-west-1.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kops.k8s.io/instancegroup=nodes,kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=172.20.0.0/16 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/ --node-ip=172.20.87.136

And the config:

admin@ip-172-20-87-136:~$ cat /lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service

[Service]
EnvironmentFile=/etc/sysconfig/kubelet
ExecStart=/usr/local/bin/kubelet "$DAEMON_ARGS"
Restart=always
RestartSec=2s
StartLimitInterval=0
KillMode=process
User=root

admin@ip-172-20-87-136:~$ cat /etc/sysconfig/kubelet
DAEMON_ARGS="--allow-privileged=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=172.20.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --feature-gates=ExperimentalCriticalPodAnnotation=true --hostname-override=ip-172-20-87-136.eu-west-1.compute.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kops.k8s.io/instancegroup=nodes,kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=172.20.0.0/16 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/ --node-ip=172.20.87.136"
HOME="/root"

I can see in the AWS console that the node and the masters have assigned secondary private IPs as expected.

kubectl describe node ip-172-20-87-136.eu-west-1.compute.internal shows:

....
Addresses:
  InternalIP:  172.20.87.136
  Hostname:    ip-172-20-87-136.eu-west-1.compute.internal
....
  Namespace                  Name                                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                      ------------  ----------  ---------------  -------------
  kube-system                aws-node-hk8t6                                            10m (0%)      0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-dns-autoscaler-787d59df8f-t7trg                      20m (1%)      0 (0%)      10Mi (0%)        0 (0%)
  kube-system                kube-dns-c58977f6c-9w7hs                                  260m (13%)    0 (0%)      110Mi (1%)       170Mi (2%)
  kube-system                kube-dns-c58977f6c-fv769                                  260m (13%)    0 (0%)      110Mi (1%)       170Mi (2%)
  kube-system                kube-proxy-ip-172-20-87-136.eu-west-1.compute.internal    100m (5%)     0 (0%)      0 (0%)           0 (0%)

I launched a pod and I can see it gets assigned an IP from the correct range:

root@test-5b77c64966-8xznp:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 3e:d8:7d:fa:13:3d
          inet addr:172.20.84.59  Bcast:172.20.84.59  Mask:255.255.255.255

kubectl describe pod test-5b77c64966-8xznp
Name:           test-5b77c64966-8xznp
Namespace:      default
Node:           ip-172-20-87-136.eu-west-1.compute.internal/172.20.87.136
Start Time:     Thu, 01 Mar 2018 17:11:24 +0100
Labels:         pod-template-hash=1633720522
                run=test
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container test
Status:         Running
IP:             172.20.84.59

I tore down the cluster now but if you have more questions please let me know.

@chrislovecnm I have rebased and ran ./hack/update-bazel.sh - fwiw I am also available in the kops-users channel in slack

dezmodue · 2018-03-01T19:32:39Z

/retest

liwenwu-amazon · 2018-03-02T15:45:01Z

@dezmodue @justinsb @chrislovecnm
I have just tested the PR. it works for me. I am able to "kubectl exec" into a Pod.

Also, I have changed to use Gossip-based cluster. Not sure why Private DNS based cluster suddenly stopped working for me now.

KashifSaadat

LGTM

chrislovecnm · 2018-03-02T21:51:10Z

@dezmodue I think we want to cherry pick this into release-1.9 branch. Do you mind?

chrislovecnm · 2018-03-02T21:51:29Z

/lgtm

k8s-ci-robot · 2018-03-02T21:52:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chrislovecnm, dezmodue

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [chrislovecnm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dezmodue · 2018-03-02T22:05:58Z

Fine by me, anything I should do?

chrislovecnm · 2018-03-02T23:02:04Z

@dezmodue just create a PR into the release branch, we are moving towards doing individual cherry picks

dezmodue · 2018-03-03T07:45:49Z

#4568 -- hope this is correct

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 9, 2018

dezmodue mentioned this pull request Feb 9, 2018

kubectl logs POD broken when using amazon-vpc-cni-k8s (kubelet registers the wrong IP) #4218

Closed

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 12, 2018

justinsb added this to the 1.9 milestone Feb 21, 2018

chrislovecnm reviewed Feb 27, 2018

View reviewed changes

chrislovecnm approved these changes Feb 27, 2018

View reviewed changes

chrislovecnm added the blocks-next label Feb 28, 2018

dezmodue added 2 commits March 1, 2018 17:47

Bind the kubelet to the local ipv4 address if the cni plugin is Amazo…

e406dbf

…nVPC - #4218

add BUILD.bazel

fcd08f1

KashifSaadat approved these changes Mar 2, 2018

View reviewed changes

chrislovecnm added the cherrypick-candidate label Mar 2, 2018

k8s-ci-robot assigned chrislovecnm Mar 2, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 2, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 2, 2018

k8s-ci-robot merged commit e634143 into kubernetes:master Mar 2, 2018

dezmodue mentioned this pull request Mar 3, 2018

Bind the kubelet to the local ipv4 address #4568

Merged

rdrgmnzs mentioned this pull request Jul 19, 2018

Allow kubelet to bind the hosts primary IP #5460

Merged

justinsb removed the cherrypick-candidate label Jul 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bind the kubelet to the local ipv4 address #4417

Bind the kubelet to the local ipv4 address #4417

dezmodue commented Feb 9, 2018

chrislovecnm commented Feb 12, 2018

bksteiny commented Feb 26, 2018

chrislovecnm Feb 27, 2018

chrislovecnm commented Feb 27, 2018

dezmodue commented Feb 27, 2018

chrislovecnm commented Feb 27, 2018

chrislovecnm commented Feb 27, 2018

justinsb commented Feb 28, 2018

dezmodue commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

liwenwu-amazon commented Feb 28, 2018 •

edited

Loading

chrislovecnm commented Feb 28, 2018

dezmodue commented Mar 1, 2018

dezmodue commented Mar 1, 2018

liwenwu-amazon commented Mar 2, 2018

KashifSaadat left a comment

chrislovecnm commented Mar 2, 2018

chrislovecnm commented Mar 2, 2018

k8s-ci-robot commented Mar 2, 2018

dezmodue commented Mar 2, 2018 •

edited

Loading

chrislovecnm commented Mar 2, 2018

dezmodue commented Mar 3, 2018

Bind the kubelet to the local ipv4 address #4417

Bind the kubelet to the local ipv4 address #4417

Conversation

dezmodue commented Feb 9, 2018

chrislovecnm commented Feb 12, 2018

bksteiny commented Feb 26, 2018

chrislovecnm Feb 27, 2018

Choose a reason for hiding this comment

chrislovecnm commented Feb 27, 2018

dezmodue commented Feb 27, 2018

chrislovecnm commented Feb 27, 2018

chrislovecnm commented Feb 27, 2018

justinsb commented Feb 28, 2018

dezmodue commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

chrislovecnm commented Feb 28, 2018

liwenwu-amazon commented Feb 28, 2018 • edited Loading

chrislovecnm commented Feb 28, 2018

dezmodue commented Mar 1, 2018

dezmodue commented Mar 1, 2018

liwenwu-amazon commented Mar 2, 2018

KashifSaadat left a comment

Choose a reason for hiding this comment

chrislovecnm commented Mar 2, 2018

chrislovecnm commented Mar 2, 2018

k8s-ci-robot commented Mar 2, 2018

dezmodue commented Mar 2, 2018 • edited Loading

chrislovecnm commented Mar 2, 2018

dezmodue commented Mar 3, 2018

liwenwu-amazon commented Feb 28, 2018 •

edited

Loading

dezmodue commented Mar 2, 2018 •

edited

Loading