Webhook didn't catch an error in spec.blockDeviceMappings[*].ebs settings #3409
Comments
|
We should remove Deletion webhooks using We should also fix this panic. |
|
Trying to reproduce this. Getting a hang instead of a crash Oddly, the instance does launch -- but why does it take so long? |
|
Ah -- was blocked on the pending snapshot |
|
Hey @andrescaroc , |
|
Noticing you're running on |
|
Ah -- looks like this is a release issue. |
No local build of Karpenter, I am using the helm chart of v0.24 (I think is the latest official release at the time of writing) |
|
I cut #3414 for this |
|
@andrescaroc can you reproduce this 100% of the time with your instructions? I am unable to. |
ellistarn
added
question
|
Labeled for closure due to inactivity in 10 days. |
|
@ellistarn I think this is still an issue: Today I tried again to use a snapshot without defining the volume size and karpenter crashes getting unrecoverable. Karpenter version: 0.27.1 Logs: AWSNodeTemplate: Karpenter gets in an unrecoverable state, trying to fix the AWSNodeTemplate resource hit this error. I have to use |
Can you confirm what @ellistarn had asked previously? Seems like he had an issue reproducing. |
Taking in consideration that is the second time that I try without setting the |
|
Ah yeah, looks like I'm getting the same issue. I'll re-open: This is my Looks like the issue is here: https://github.com/aws/karpenter/blob/main/pkg/providers/instancetype/types.go#L150 and then in https://github.com/aws/karpenter/blob/main/pkg/providers/instancetype/types.go#L176 here. When I changed my amiFamily to use AL2, I didn't see the crash, but I did still get an error: Going to re-open so we can fix this. |
|
It looks like for EBS root volumes: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html
And our validation logic just checks if one of |
|
Thanks @njtran, I think you catched the error. In my case I use bottlerocketOS which by default comes with two volumes (root and data), I want to use a snapshot for the data volume Based on the aws documentation I should be able to provide an EBS Block device mapping without having to specify the volume size. |
@njtran about the error that you are getting for the snapshot itself, I will suggest you to test with a snapshot of your own, I used a random SnapshotID in my description. A snapshot of the data volume of a Bottlerocket OS instance will do the job. Otherwise you may loose time trying to debug something not related with the main issue. |
Version
Karpenter Version: v0.24.0
Kubernetes Version: v1.23.0
Expected Behavior
If there is an error in the settings of an
AWSNodeTemplateit should be cached by the karpenter validating webhook.In my case It seems karpenter is forcing me to define an
ebs.volumeSizewhen I already defined anebs.snapshotID.However, according the aws API documentation is only required one of the two, and is possible to define both under a condition:
Actual Behavior
I was trying to deploy a Provisioner refering to an
AWSNodeTemplatewith missing required fields (I was not aware of) in theblockDeviceMappingssection , the validating webhook let it pass and be deployed.Right away the karpenter pods start to crash letting the karpenter useless.
karpenter pods watch:
Steps to Reproduce the Problem
Define a provisioner that refers to an AWSNodeTemplate as follows:
Define an AWSNodeTemplate that will use a snapshot as follows:
I am going to focus in the blockDeviceMappings section that was my case, but might be happening in other nested sections under
specDeploy both resources:
No complains shown by the karpenter webhook.
Watch the karpenter pods:
Karpenter pods shoud start to restart and eventually crash both
You will not be able to do any other operation like editing karpenter resources (the node template) or delete it because the karpenter webhook won't be available.
To solve it you need to rollout restart the karpenter deployment and patch the faulty AWSNodeTemplate right away, previous they start to crash again.
Resource Specs and Logs
Logs were not useful to find the issue:
2023-02-16T19:04:04.066Z INFO controller Starting workers {"commit": "8c27519-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "worker count": 10} 2023-02-16T19:04:04.077Z INFO controller Starting workers {"commit": "8c27519-dirty", "controller": "inflightchecks", "controllerGroup": "", "controllerKind": "Node", "worker count": 10} 2023-02-16T19:04:04.077Z INFO controller Starting workers {"commit": "8c27519-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10} 2023-02-16T19:04:04.128Z DEBUG controller.awsnodetemplate discovered security groups {"commit": "8c27519-dirty", "awsnodetemplate": "bottlerocket", "security-groups": ["sg-0830faredacted366"]} 2023-02-16T19:04:04.953Z DEBUG controller.deprovisioning discovered EC2 instance types {"commit": "8c27519-dirty", "instance-type-count": 505} 2023-02-16T19:04:05.092Z DEBUG controller.deprovisioning discovered EC2 instance types zonal offerings for subnets {"commit": "8c27519-dirty", "subnet-selector": "{\"Name\":\"*Private*\",\"karpenter.sh/discovery\":\"my-cluster\"}"} 2023-02-16T19:04:05.118Z INFO controller.deprovisioning stopping controller {"commit": "8c27519-dirty"} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1c521fb] goroutine 750 [running]: github.com/aws/karpenter/pkg/cloudprovider.computeCapacity({0x2ce89d8, 0xc001dd2c90}, 0xc000ce4c00, {0x2cec768, 0xc0020b0b50}, {0xc0020b0b48, 0x1, 0x1}, 0xc000056023?) github.com/aws/karpenter/pkg/cloudprovider/instancetype.go:134 +0x1db github.com/aws/karpenter/pkg/cloudprovider.NewInstanceType({0x2ce89d8, 0xc001dd2c90}, 0xc000ce4c00, 0x0?, {0xc000056023, 0xc}, 0xc0016183c0, {0xc00340af00, 0x6, 0x8}) github.com/aws/karpenter/pkg/cloudprovider/instancetype.go:58 +0x315 github.com/aws/karpenter/pkg/cloudprovider.(*InstanceTypeProvider).List(0xc000c2fc20, {0x2ce89d8, 0xc001dd2c90}, 0x0?, 0xc0016183c0) github.com/aws/karpenter/pkg/cloudprovider/instancetypes.go:112 +0x425 github.com/aws/karpenter/pkg/cloudprovider.(*CloudProvider).GetInstanceTypes(0xc000d6f3b0, {0x2ce89d8, 0xc001dd2c90}, 0xc0010a09c8) github.com/aws/karpenter/pkg/cloudprovider/cloudprovider.go:181 +0x9a github.com/aws/karpenter-core/pkg/cloudprovider/metrics.(*decorator).GetInstanceTypes(0xc000d38190, {0x2ce89d8, 0xc001dd2c90}, 0x13?) github.com/aws/[email protected]/pkg/cloudprovider/metrics/cloudprovider.go:86 +0x199 github.com/aws/karpenter-core/pkg/controllers/deprovisioning.buildProvisionerMap({0x2ce89d8, 0xc001dd2c90}, {0x2cf3fe8, 0xc000d7ef00}, {0x2cec648, 0xc000d38190}) github.com/aws/[email protected]/pkg/controllers/deprovisioning/helpers.go:252 +0x1fe github.com/aws/karpenter-core/pkg/controllers/deprovisioning.candidateNodes({0x2ce89d8, 0xc001dd2c90}, 0xc001dd2cc0?, {0x2cf3fe8, 0xc000d7ef00}, {0x2cede30, 0x419c5d0}, {0x2cec648?, 0xc000d38190?}, 0xc0023f1d50) github.com/aws/[email protected]/pkg/controllers/deprovisioning/helpers.go:162 +0x7c github.com/aws/karpenter-core/pkg/controllers/deprovisioning.(*Controller).Reconcile(0xc00042e0e0, {0x2ce89d8, 0xc001dd2c90}, {{{0x0?, 0x0?}, {0x404d6c?, 0x0?}}}) github.com/aws/[email protected]/pkg/controllers/deprovisioning/controller.go:110 +0x179 github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).reconcile(0xc000d6f950, {0x2ce89d8, 0xc001dd2c90}) github.com/aws/[email protected]/pkg/operator/controller/singleton.go:108 +0xb9 github.com/aws/karpenter-core/pkg/operator/controller.(*Singleton).Start(0xc000d6f950, {0x2ce8930, 0xc000127340}) github.com/aws/[email protected]/pkg/operator/controller/singleton.go:99 +0x205 sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0xc000d28860) sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:219 +0xdb created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile sigs.k8s.io/[email protected]/pkg/manager/runnable_group.go:203 +0x1adCommunity Note
The text was updated successfully, but these errors were encountered: