Kops update with private networking is trying to change NatGateway #1073

chrislovecnm · 2016-12-06T20:25:03Z

Bug description

Our new integration tests are showing that kops update is actually changing aws components. It is my understanding that an update preps the cluster for an update and does not touch aws resources.

Steps to recreate

You can recreate by running the make e2e target in my PR #1064. You will need the following env vars to run the tests:

       GIT_VER=git-$(git describe --always)
        KOPS_TEST_IMAGE=image-fixed-for-private-networking \
        KOPS_TEST_DOMAIN=test-domain.com \
        KOPS_TEST_NODEUP_URL="https://s3-us-west-1.amazonaws.com/your-nodeup-bucket/kops/${GIT_VER}/linux/amd64/nodeup" \
        KOPS_TEST_K8S_VERSION="v1.4.5"

What the tests actually do

The sequence the test is doing:

create s3 state store bucket
create cluster
validate cluster
upgrade cluster
update cluster // acting weird here
rolling-update cluster
validate cluster
delete cluster
delete s3 bucket

Logs from testing

I1206 13:15:09.372166   74432 reflect.go:56] Calling method CheckChanges on *awstasks.Route
I1206 13:15:09.372200   74432 natgateway.go:146] Waiting for NAT Gateway "nat-04716abd24496fe11" to be available
I1206 13:15:09.372319   74432 request_logger.go:45] AWS request: ec2/DescribeNatGateways
I1206 13:15:09.374007   74432 route.go:97] found route matching cidr 0.0.0.0/0
I1206 13:15:09.374105   74432 changes.go:80] Field changed "NatGateway" actual="{<nil> *awstasks.ElasticIP null *awstasks.Subnet null 0xc4205c2ac8}" expected="{0xc42054c3c0 *awstasks.ElasticIP {\"Name\":\"eu-west-1c.kops-test-william-slytherin.dev.datapipe.io\",\"ID\":\"eipalloc-2095bc44\",\"PublicIP\":null,\"Subnet\":{\"Name\":\"utility-eu-west-1c.kops-test-william-slytherin.dev.datapipe.io\",\"ID\":\"subnet-161d4e60\",\"VPC\":{\"Name\":\"kops-test-william-slytherin.dev.datapipe.io\",\"ID\":\"vpc-06e3a062\",\"CIDR\":\"172.20.0.0/16\",\"EnableDNSHostnames\":true,\"EnableDNSSupport\":true,\"Shared\":false},\"AvailabilityZone\":\"eu-west-1c\",\"CIDR\":\"172.20.96.0/19\",\"Shared\":false}} *awstasks.Subnet {\"Name\":\"utility-eu-west-1c.kops-test-william-slytherin.dev.datapipe.io\",\"ID\":\"subnet-161d4e60\",\"VPC\":{\"Name\":\"kops-test-william-slytherin.dev.datapipe.io\",\"ID\":\"vpc-06e3a062\",\"CIDR\":\"172.20.0.0/16\",\"EnableDNSHostnames\":true,\"EnableDNSSupport\":true,\"Shared\":false},\"AvailabilityZone\":\"eu-west-1c\",\"CIDR\":\"172.20.96.0/19\",\"Shared\":false} 0xc420012ac8}"
I1206 13:15:09.374149   74432 reflect.go:56] Calling method CheckChanges on *awstasks.Route
I1206 13:15:09.374192   74432 natgateway.go:146] Waiting for NAT Gateway "nat-0095a67a61f8a175b" to be available
I1206 13:15:09.374347   74432 request_logger.go:45] AWS request: ec2/DescribeNatGateways
W1206 13:15:09.588753   74432 executor.go:108] error running task "route/private-eu-west-1a.kops-test-william-slytherin.dev.datapipe.io" (-8.016440218s remaining to succeed): error waiting for NAT Gateway "nat-0ef857130a4498075" to be available: ResourceNotReady: failed waiting for successful resource state
W1206 13:15:09.588776   74432 executor.go:108] error running task "route/private-eu-west-1b.kops-test-william-slytherin.dev.datapipe.io" (-8.01640953s remaining to succeed): error waiting for NAT Gateway "nat-04716abd24496fe11" to be available: ResourceNotReady: failed waiting for successful resource state
W1206 13:15:09.588788   74432 executor.go:108] error running task "route/private-eu-west-1c.kops-test-william-slytherin.dev.datapipe.io" (-8.016390656s remaining to succeed): error waiting for NAT Gateway "nat-0095a67a61f8a175b" to be available: ResourceNotReady: failed waiting for successful resource state
I1206 13:15:09.588824   74432 executor.go:123] No progress made, sleeping before retrying 3 failed task(s)
I1206 13:15:19.591852   74432 context.go:77] deleting temp dir: "/var/folders/7g/6zkk5fjn67x2rpg64bv3n1pr0000gn/T/deploy256519845"

error running tasks: deadline exceeded executing task route/private-eu-west-1b.kops-test-william-slytherin.dev.datapipe.io. Example error: error waiting for NAT Gateway "nat-04716abd24496fe11" to be available: ResourceNotReady: failed waiting for successful resource state

Open question

Not sure if #1072 fixed this, but I am thinking no.

The text was updated successfully, but these errors were encountered:

justinsb · 2016-12-07T01:19:20Z

I don't see a change being made to AWS in the logs?

chrislovecnm · 2016-12-07T07:31:14Z

kops create cluster <- HA, Private Networking, Weave, v1.4.6
kops edit cluster <- change version to v1.4.5
kops update cluster <- Not expecting the following:

$ KOPS_STATE_STORE="s3://oscar-ai-k8s-dev" kops update cluster
Using cluster from kubectl context: liea-ginny.dev.datapipe.io

I1207 02:25:50.126443    2556 populate_cluster_spec.go:234] Defaulting DNS zone to: Z2PKVP3BDFE581
I1207 02:25:51.070991    2556 executor.go:91] Tasks: 0 done / 104 total; 32 can run
I1207 02:25:51.991981    2556 executor.go:91] Tasks: 32 done / 104 total; 21 can run
I1207 02:25:52.637032    2556 executor.go:91] Tasks: 53 done / 104 total; 32 can run
I1207 02:25:53.404287    2556 executor.go:91] Tasks: 85 done / 104 total; 11 can run
I1207 02:25:53.785217    2556 executor.go:91] Tasks: 96 done / 104 total; 8 can run
I1207 02:25:54.111160    2556 executor.go:91] Tasks: 104 done / 104 total; 0 can run
Will create resources:
  NatGateway          	ngw/eu-west-1b.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1b.liea-ginny.dev.datapipe.io","ID":"eipalloc-a9cfe7cd","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1b.liea-ginny.dev.datapipe.io","ID":"subnet-395d055d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1b","CIDR":"172.20.64.0/19","Shared":false}}
  	Subnet              	id:subnet-395d055d

  NatGateway          	ngw/eu-west-1c.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1c.liea-ginny.dev.datapipe.io","ID":"eipalloc-18cce47c","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1c.liea-ginny.dev.datapipe.io","ID":"subnet-eb491a9d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1c","CIDR":"172.20.96.0/19","Shared":false}}
  	Subnet              	id:subnet-eb491a9d

  NatGateway          	ngw/eu-west-1a.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1a.liea-ginny.dev.datapipe.io","ID":"eipalloc-ffcbe39b","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1a.liea-ginny.dev.datapipe.io","ID":"subnet-980399c0","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}}
  	Subnet              	id:subnet-980399c0

Will modify resources:
  ElasticIP           	elasticIP/eu-west-1c.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1c.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1b.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1b.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1a.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1a.liea-ginny.dev.datapipe.io

  LaunchConfiguration 	launchConfiguration/master-eu-west-1c.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/nodes.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/master-eu-west-1a.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/master-eu-west-1b.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  DNSName             	dnsName/api.liea-ginny.dev.datapipe.io
  	TargetLoadBalancer  	 <nil> -> id:api-liea-ginny

  Route               	route/private-eu-west-1c.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-08e89472f783e975e -> id:<nil>

  Route               	route/private-eu-west-1b.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-058c9eb8edb97a8f2 -> id:<nil>

  Route               	route/private-eu-west-1a.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-0b24d9deeba3e09c7 -> id:<nil>

Must specify --yes to apply changes

^ specifically

	TargetLoadBalancer  	 <nil> -> id:api-liea-ginny

  Route               	route/private-eu-west-1c.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-08e89472f783e975e -> id:<nil>

  Route               	route/private-eu-west-1b.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-058c9eb8edb97a8f2 -> id:<nil>

  Route               	route/private-eu-west-1a.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-0b24d9deeba3e09c7 -> id:<nil>

Goind to see if the update changes the NatGateway....

chrislovecnm · 2016-12-07T07:32:08Z

yah this smells like a bug:

Will create resources:
  NatGateway          	ngw/eu-west-1a.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1a.liea-ginny.dev.datapipe.io","ID":"eipalloc-ffcbe39b","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1a.liea-ginny.dev.datapipe.io","ID":"subnet-980399c0","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}}
  	Subnet              	id:subnet-980399c0

  NatGateway          	ngw/eu-west-1b.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1b.liea-ginny.dev.datapipe.io","ID":"eipalloc-a9cfe7cd","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1b.liea-ginny.dev.datapipe.io","ID":"subnet-395d055d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1b","CIDR":"172.20.64.0/19","Shared":false}}
  	Subnet              	id:subnet-395d055d

  NatGateway          	ngw/eu-west-1c.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1c.liea-ginny.dev.datapipe.io","ID":"eipalloc-18cce47c","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1c.liea-ginny.dev.datapipe.io","ID":"subnet-eb491a9d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1c","CIDR":"172.20.96.0/19","Shared":false}}
  	Subnet              	id:subnet-eb491a9d

Will modify resources:
  ElasticIP           	elasticIP/eu-west-1b.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1b.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1a.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1a.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1c.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1c.liea-ginny.dev.datapipe.io

  LaunchConfiguration 	launchConfiguration/nodes.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/master-eu-west-1a.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/master-eu-west-1c.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  LaunchConfiguration 	launchConfiguration/master-eu-west-1b.masters.liea-ginny.dev.datapipe.io
  	UserData
  	                    	...
  	                    	  set -o pipefail

  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.4.1/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://s3-us-west-1.amazonaws.com/kops-devel/kops/git-270bf37/linux/amd64/nodeup
  	                    	  NODEUP_HASH=

  	                    	...
  	                    	  cat > kube_env.yaml << __EOF_KUBE_ENV
  	                    	  Assets:
  	                    	+ - fffb150fd3dfce79c9bed7641e39e2c228ffae1e@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubelet
  	                    	- - 7d70e090951486cae52d9a82b7aaf5056f84f8ed@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubelet
  	                    	+ - d69371cdb4091331efb421b8df0d7f7148a3dc05@https://storage.googleapis.com/kubernetes-release/release/v1.4.5/bin/linux/amd64/kubectl
  	                    	- - 9adcd120fdb7ad6e64c061e56a05fefc12e9618b@https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl
  	                    	  - 19d49f7b2b99cd2493d5ae0ace896c64e289ccbb@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz
  	                    	  ClusterName: liea-ginny.dev.datapipe.io
  	                    	...


  DNSName             	dnsName/api.liea-ginny.dev.datapipe.io
  	TargetLoadBalancer  	 <nil> -> id:api-liea-ginny

  Route               	route/private-eu-west-1a.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-0b24d9deeba3e09c7 -> id:<nil>

  Route               	route/private-eu-west-1c.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-08e89472f783e975e -> id:<nil>

  Route               	route/private-eu-west-1b.liea-ginny.dev.datapipe.io
  	NatGateway          	 id:nat-058c9eb8edb97a8f2

chrislovecnm · 2016-12-07T16:00:43Z

Simple steps to recreate:

kops create cluster with our dev script (set k8s version to 1.4.5)
kops edit cluster change k8s version to 1.4.6
kops upgrade cluster you will see that it is try to upgrade network components that should not be upgraded
kops upgrade cluster --yes <- this will actually upgrade components: DNSNAme, Nategatweay stuff, routes, elastic ips

Will create resources:
  NatGateway          	ngw/eu-west-1a.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1a.liea-ginny.dev.datapipe.io","ID":"eipalloc-ffcbe39b","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1a.liea-ginny.dev.datapipe.io","ID":"subnet-980399c0","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1a","CIDR":"172.20.32.0/19","Shared":false}}
  	Subnet              	id:subnet-980399c0

  NatGateway          	ngw/eu-west-1b.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1b.liea-ginny.dev.datapipe.io","ID":"eipalloc-a9cfe7cd","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1b.liea-ginny.dev.datapipe.io","ID":"subnet-395d055d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1b","CIDR":"172.20.64.0/19","Shared":false}}
  	Subnet              	id:subnet-395d055d

  NatGateway          	ngw/eu-west-1c.liea-ginny.dev.datapipe.io
  	ElasticIp           	{"Name":"eu-west-1c.liea-ginny.dev.datapipe.io","ID":"eipalloc-18cce47c","PublicIP":null,"Subnet":{"Name":"utility-eu-west-1c.liea-ginny.dev.datapipe.io","ID":"subnet-eb491a9d","VPC":{"Name":"liea-ginny.dev.datapipe.io","ID":"vpc-e4327080","CIDR":"172.20.0.0/16","EnableDNSHostnames":true,"EnableDNSSupport":true,"Shared":false},"AvailabilityZone":"eu-west-1c","CIDR":"172.20.96.0/19","Shared":false}}
  	Subnet              	id:subnet-eb491a9d

Will modify resources:
  ElasticIP           	elasticIP/eu-west-1b.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1b.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1a.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1a.liea-ginny.dev.datapipe.io

  ElasticIP           	elasticIP/eu-west-1c.liea-ginny.dev.datapipe.io
  	Name                	 <nil> -> eu-west-1c.liea-ginny.dev.datapipe.io

chrislovecnm · 2016-12-19T01:38:45Z

Need to test against We have PR inbound thay may help with this ... I am looking through issues to test against #1183 - no promises but we need to test.

chrislovecnm · 2016-12-19T01:38:54Z

This should be closed ;)

chrislovecnm · 2016-12-19T22:52:16Z

Closing this.

chrislovecnm added the P0 label Dec 6, 2016

chrislovecnm added this to the 1.4.2 milestone Dec 6, 2016

chrislovecnm assigned chrislovecnm and krisnova Dec 6, 2016

chrislovecnm added bug and removed P0 labels Dec 19, 2016

chrislovecnm mentioned this issue Dec 19, 2016

Umbrella List of Test Issues and Test Cases #1206

Closed

27 tasks

chrislovecnm closed this as completed Dec 19, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kops update with private networking is trying to change NatGateway #1073

Kops update with private networking is trying to change NatGateway #1073

chrislovecnm commented Dec 6, 2016 •

edited

Loading

justinsb commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 19, 2016

chrislovecnm commented Dec 19, 2016

chrislovecnm commented Dec 19, 2016

Kops update with private networking is trying to change NatGateway #1073

Kops update with private networking is trying to change NatGateway #1073

Comments

chrislovecnm commented Dec 6, 2016 • edited Loading

Bug description

Steps to recreate

What the tests actually do

Logs from testing

Open question

justinsb commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 7, 2016

chrislovecnm commented Dec 19, 2016

chrislovecnm commented Dec 19, 2016

chrislovecnm commented Dec 19, 2016

chrislovecnm commented Dec 6, 2016 •

edited

Loading