Nginx crashes - conditions: unable to find real backend for alternative backend while using canary feature and duplicate ingress objects exist in cluster.

[christianjoun]

NGINX Ingress controller version:

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.6", GitCommit:"72c30166b2105cd7d3350f2c28a219e6abcd79eb", GitTreeState:"clean", BuildDate:"2020-01-18T23:31:31Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T20:52:22Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: aws
• OS (e.g. from /etc/os-release): PRETTY_NAME="Debian GNU/Linux buster/sid"
• Kernel (e.g. uname -a): Linux nginx-ingress-internal-ieng-controller-7f74f45cc9-5n6d4 4.15.0-70-generic make test-e2e fails on push #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
in ns ieng-training, an ingress with 0% canary pointing to a working backend
in ns scancloud-ci, an ingress with 0% canary pointing to a working backend
in ns scancloud-ci, an ingress pointing to a backend with no endpoints
nginx crashed.

please include exact error messages if you can:

W0505 22:47:37.574531       6 controller.go:797] Service "scancloud-ci/scancloud-connector" does not have any active Endpoint.
W0505 22:47:37.576506       6 controller.go:1218] unable to find real backend for alternative backend ieng-training-scancloud-connector-canary-443. Deleting.
E0505 22:47:37.576642       6 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1209
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1209
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:529
/go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:122
/go/src/k8s.io/ingress-nginx/internal/task/queue.go:129
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/k8s.io/ingress-nginx/internal/task/queue.go:61
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x12f20e2]
goroutine 157 [running]:
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x14a6640, 0x25ff390)
    /usr/local/go/src/runtime/panic.go:522 +0x1b5
k8s.io/ingress-nginx/internal/ingress/controller.canMergeBackend(...)
    /go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1122
k8s.io/ingress-nginx/internal/ingress/controller.mergeAlternativeBackends(0xc0008fd500, 0xc0332ba090, 0xc0333aa720)
    /go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:1209 +0x142
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).getBackendServers(0xc000437260, 0xc0063fb000, 0x1d5, 0x200, 0x2636cf0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:529 +0x1804
k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).syncIngress(0xc000437260, 0x153fbe0, 0xc00fccfbe0, 0xc02e3d22fa, 0x564009a81293)
    /go/src/k8s.io/ingress-nginx/internal/ingress/controller/controller.go:122 +0xbc
k8s.io/ingress-nginx/internal/task.(*Queue).worker(0xc0004fb170)
    /go/src/k8s.io/ingress-nginx/internal/task/queue.go:129 +0x312
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000e5bfa8)
    /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00fb4dfa8, 0x3b9aca00, 0x0, 0x1709a01, 0xc0004cad20)
    /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
    /go/src/k8s.io/ingress-nginx/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
k8s.io/ingress-nginx/internal/task.(*Queue).Run(0xc0004fb170, 0x3b9aca00, 0xc0004cad20)
    /go/src/k8s.io/ingress-nginx/internal/task/queue.go:61 +0x6c
created by k8s.io/ingress-nginx/internal/ingress/controller.(*NGINXController).Start
    /go/src/k8s.io/ingress-nginx/internal/ingress/controller/nginx.go:314 +0x474

What you expected to happen:

What do you think went wrong?:

nil check is not in place and duplicate ingresses in two namespaces, related to canary feature.

How to reproduce it:

We have had difficulty reproducing this issue on another cluster, and believe it is order dependent. However, I have included the process and yamls we expected to reproduce the issue but could not:

in ns ieng-training, an ingress with 0% canary pointing to a working backend
in ns scancloud-ci, an ingress with 0% canary pointing to a working backend
in ns scancloud-ci, an ingress pointing to a backend with no endpoints

Create two namespaces "development" & "production"

Run kubectl apply -f ./reproduce-nginx-bug.yaml.txt & Scale down the deployment in production to 0 to replicate no endpoints.

note: You will need to supply an container image that can respond to https traffic on port 8443.

/kind bug
reproduce-nginx-bug.yaml.txt

[aledbf]

@christianjoun what version of the ingress controller are you using?

[christianjoun]

release 0.24.1 last commit approx april 23 2019

### 0.24.1

**Image:** `quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1`

_Changes:_

- [X] [#3990](https://github.com/kubernetes/ingress-nginx/pull/3990) Fix dynamic cert issue with default-ssl-certificate
- [X] [#3980](https://github.com/kubernetes/ingress-nginx/pull/3980) Refactor isIterable
- [X] [#4000](https://github.com/kubernetes/ingress-nginx/pull/4000) Dynamic ssl improvements
- [X] [#4007](https://github.com/kubernetes/ingress-nginx/pull/4007) do not create empty access_by_lua_block

#3991

[aledbf]

release 0.24.1 last commit approx april 23 2019

Update to the latest version, 0.32.0. Too many changes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[christianjoun]

/remove-lifecycle-rotten

[aledbf]

Closing. Please reopen if the issue persists after updating to the latest version 0.35.0