invalid memory address or nil pointer dereference

[maxirus]

Recently upgraded Kubernetes to v1.23.4+k3s1. Now when a new Volume request comes in, I am receiving the error:

E0520 19:18:14.718931       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 145 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15ce420, 0x22c5770)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x15ce420, 0x22c5770)
	/home/travis/.gimme/versions/go1.15.12.linux.amd64/src/runtime/panic.go:969 +0x1b9
github.com/nmaupu/freenas-provisioner/provisioner.(*freenasProvisioner).Provision(0xc00000c4c0, 0x1983940, 0xc00011bc80, 0xc000678000, 0xc000042240, 0x28, 0xc00081a000, 0x0, 0x5d, 0x1, ...)
	/home/travis/gopath/src/github.com/nmaupu/freenas-provisioner/provisioner/provisioner.go:245 +0x43
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).provisionClaimOperation(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0xc00081a000, 0xc000042101, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1404 +0xd2e
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).syncClaim(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0x1771fc0, 0xc00081a000, 0xc0004e4070, 0x1)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1090 +0x10e
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).syncClaimHandler(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0xc00076c120, 0x24, 0x10, 0x10)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1058 +0xca
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).processNextClaimWorkItem.func1(0xc0001b8f00, 0xc00054c618, 0x1567fe0, 0xc0004e4060, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:959 +0x125
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).processNextClaimWorkItem(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:981 +0x71
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).runClaimWorker(0xc0001b8f00, 0x1983940, 0xc00011bc80)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:927 +0x3f
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).Run.func1.2()
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:883 +0x3c
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00033c0a0)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00033c0a0, 0x1947f00, 0xc000542030, 0x1, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00033c0a0, 0x3b9aca00, 0x0, 0x1, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00033c0a0, 0x3b9aca00, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).Run.func1
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:883 +0x4af
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x145a9e3]
goroutine 145 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x15ce420, 0x22c5770)
	/home/travis/.gimme/versions/go1.15.12.linux.amd64/src/runtime/panic.go:969 +0x1b9
github.com/nmaupu/freenas-provisioner/provisioner.(*freenasProvisioner).Provision(0xc00000c4c0, 0x1983940, 0xc00011bc80, 0xc000678000, 0xc000042240, 0x28, 0xc00081a000, 0x0, 0x5d, 0x1, ...)
	/home/travis/gopath/src/github.com/nmaupu/freenas-provisioner/provisioner/provisioner.go:245 +0x43
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).provisionClaimOperation(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0xc00081a000, 0xc000042101, 0x0, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1404 +0xd2e
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).syncClaim(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0x1771fc0, 0xc00081a000, 0xc0004e4070, 0x1)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1090 +0x10e
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).syncClaimHandler(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0xc00076c120, 0x24, 0x10, 0x10)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:1058 +0xca
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).processNextClaimWorkItem.func1(0xc0001b8f00, 0xc00054c618, 0x1567fe0, 0xc0004e4060, 0x0, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:959 +0x125
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).processNextClaimWorkItem(0xc0001b8f00, 0x1983940, 0xc00011bc80, 0x0)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:981 +0x71
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).runClaimWorker(0xc0001b8f00, 0x1983940, 0xc00011bc80)
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:927 +0x3f
sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).Run.func1.2()
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:883 +0x3c
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00033c0a0)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00033c0a0, 0x1947f00, 0xc000542030, 0x1, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00033c0a0, 0x3b9aca00, 0x0, 0x1, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc00033c0a0, 0x3b9aca00, 0xc00053c480)
	/home/travis/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90 +0x4d
created by sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller.(*ProvisionController).Run.func1
	/home/travis/gopath/pkg/mod/sigs.k8s.io/sig-storage-lib-external-provisioner/[email protected]/controller/controller.go:883 +0x4af

Environment:

• Kubernetes: v1.23.4+k3s1
• FreeNAS: 11.3-U5
• freenas-provisioner: v2.7

A shame as this has been working great for a number of years.

[maxirus]

Ok, digging into the error a little more, I found a reference to provisioner.go#L245 which appears to get the StorageClass name. If I leave my storage class name empty (so default), it works. Setting it to freenas-nfs (which is the name of storageClass for freenas-provisioner) causes it to break it seems.

There may also be some issues with k3s' local-path storage messing with this as well which I disabled as well.