Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subnet: handle unknown state in delete event #1000

Merged
merged 1 commit into from
Jul 9, 2018

Conversation

ggaaooppeenngg
Copy link
Contributor

@ggaaooppeenngg ggaaooppeenngg commented May 31, 2018

To fix panic (see log below), add a handler for
cache.DeletedFinalStateUnknown.

/usr/local/go/src/runtime/asm_amd64.s:2197
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node

goroutine 19 [running]:
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).handleAddLeaseEvent(0xc420014f00, 0x1, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:176 +0x2d3
github.com/coreos/flannel/subnet/kube.newKubeSubnetManager.func4(0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:165 +0x47
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(0xc4202ba6e0, 0xc4202ba6f0, 0xc4202ba700, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:206 +0x49
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc42018c780, 0x17e38a0, 0xc4204f61c0)
	<autogenerated>:56 +0x73
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.NewIndexerInformer.func1(0x1792520, 0xc4204f6680, 0x1792520, 0xc4204f6680)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:342 +0x519
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4203120c0, 0xc4203aede0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x27e
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc420067180)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4205fcf70)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420865f70, 0x3b9aca00, 0x0, 0x166ac01, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4205fcf70, 0x3b9aca00, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc420067180, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x237
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).Run(0xc420014f00, 0x7f340ee4d3b0, 0xc4200161a8)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:284 +0x8a
created by github.com/coreos/flannel/subnet/kube.NewSubnetManager
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:128 +0x592

Signed-off-by: Peng Gao [email protected]

Description

Todos

  • Tests
  • Documentation
  • Release note

Release Note

None required

To fix panic (see log below), add a handler for
cache.DeletedFinalStateUnknown.

```
/usr/local/go/src/runtime/asm_amd64.s:2197
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node

goroutine 19 [running]:
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).handleAddLeaseEvent(0xc420014f00, 0x1, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:176 +0x2d3
github.com/coreos/flannel/subnet/kube.newKubeSubnetManager.func4(0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:165 +0x47
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(0xc4202ba6e0, 0xc4202ba6f0, 0xc4202ba700, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:206 +0x49
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc42018c780, 0x17e38a0, 0xc4204f61c0)
	<autogenerated>:56 +0x73
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.NewIndexerInformer.func1(0x1792520, 0xc4204f6680, 0x1792520, 0xc4204f6680)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:342 +0x519
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4203120c0, 0xc4203aede0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x27e
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc420067180)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4205fcf70)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420865f70, 0x3b9aca00, 0x0, 0x166ac01, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4205fcf70, 0x3b9aca00, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc420067180, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x237
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).Run(0xc420014f00, 0x7f340ee4d3b0, 0xc4200161a8)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:284 +0x8a
created by github.com/coreos/flannel/subnet/kube.NewSubnetManager
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:128 +0x592
```

Signed-off-by: Peng Gao <[email protected]>
Copy link
Contributor

@tomdee tomdee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tomdee tomdee merged commit c7ee1fc into flannel-io:master Jul 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants