Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix panic in termination flow #205

Merged

Conversation

jonathan-innis
Copy link
Member

@jonathan-innis jonathan-innis commented Feb 6, 2023

Fixes #

Description

  • Termination could panic with the following error when the Patch() update results in the apiserver immediately return back an object that doesn't contain a deletionTimestamp set inside of it (because the object was quickly removed)
2023-02-06T21:35:18.371Z	INFO	controller	Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference	{"commit": "afe64e9", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "Node": {"name":"ip-192-168-112-11.us-east-2.compute.internal"}, "namespace": "", "name": "ip-192-168-112-11.us-east-2.compute.internal", "reconcileID": "4e585e34-4d43-412e-9f77-c5910f911e5f"}
        panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        	panic: runtime error: invalid memory address or nil pointer dereference
        [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1f66d8e]
        
        goroutine 1021 [running]:
        [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Reconcile.func1()
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118) +0x1f4
        panic({0x2362b80, 0x413de40})
        	runtime/panic.go:884 +0x212
        [github.com/aws/karpenter-core/pkg/controllers/machine/terminator.(*Terminator](http://github.com/aws/karpenter-core/pkg/controllers/machine/terminator.(*Terminator)).TerminateNode(0xc000460c40, {0x2ce8a38, 0xc001d22a80}, 0xc0038b18c0)
        	[github.com/aws/[email protected]/pkg/controllers/machine/terminator/terminator.go:114](http://github.com/aws/[email protected]/pkg/controllers/machine/terminator/terminator.go:114) +0x28e
        [github.com/aws/karpenter-core/pkg/controllers/termination.(*Controller](http://github.com/aws/karpenter-core/pkg/controllers/termination.(*Controller)).Finalize(0xc000bfadb0, {0x2ce8a38, 0xc001d22a80}, 0xc0038b18c0)
        	[github.com/aws/[email protected]/pkg/controllers/termination/controller.go:78](http://github.com/aws/[email protected]/pkg/controllers/termination/controller.go:78) +0x3d1
        [github.com/aws/karpenter-core/pkg/operator/controller.(*typedDecorator[...]](http://github.com/aws/karpenter-core/pkg/operator/controller.(*typedDecorator[...])).Reconcile(0xc0004d0960, {0x2ce8a38, 0xc001d229f0}, {{{0x0, 0xc005bc0200?}, {0xc006ec93e0?, 0x7fb82d4ebaf0?}}})
        	[github.com/aws/[email protected]/pkg/operator/controller/typed.go:77](http://github.com/aws/[email protected]/pkg/operator/controller/typed.go:77) +0x669
        [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Reconcile(0x2ce8990?, {0x2ce8a38?, 0xc001d229f0?}, {{{0x0?, 0x2670220?}, {0xc006ec93e0?, 0x404cf4?}}})
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121) +0xc8
        [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).reconcileHandler(0xc000f53b80, {0x2ce8990, 0xc000cb6dc0}, {0x2465aa0?, 0xc002db7a60?})
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320) +0x33c
        [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).processNextWorkItem(0xc000f53b80, {0x2ce8990, 0xc000cb6dc0})
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273) +0x1d9
        [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Start.func2.2()
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234) +0x85
        created by [sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller](http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller)).Start.func2
        	[sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230](http://sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230) +0x333

How was this change tested?

make presubmit

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@jonathan-innis jonathan-innis requested a review from a team as a code owner February 6, 2023 23:05
@jonathan-innis jonathan-innis requested a review from njtran February 6, 2023 23:05
@jonathan-innis jonathan-innis force-pushed the fix-termination-panic branch 2 times, most recently from 11460fe to e421602 Compare February 6, 2023 23:07
@coveralls
Copy link

coveralls commented Feb 6, 2023

Pull Request Test Coverage Report for Build 4108802140

  • 1 of 4 (25.0%) changed or added relevant lines in 2 files are covered.
  • 14 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.2%) to 78.312%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/controllers/termination/controller.go 0 3 0.0%
Files with Coverage Reduction New Missed Lines %
pkg/test/cachesyncingclient.go 2 73.91%
pkg/controllers/deprovisioning/controller.go 3 76.25%
pkg/controllers/deprovisioning/events/events.go 9 80.43%
Totals Coverage Status
Change from base Build 4108449011: -0.2%
Covered Lines: 6236
Relevant Lines: 7963

💛 - Coveralls

@jonathan-innis jonathan-innis enabled auto-merge (squash) February 6, 2023 23:21
Copy link
Contributor

@bwagner5 bwagner5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jonathan-innis jonathan-innis merged commit 6abab23 into kubernetes-sigs:main Feb 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants