Get cluster that doesn't exist gives 502 instead of 404

[edda]

Trying to get a cluster that doesn't exist anymore (/api/v1/clusters/bla) results in a 502 Bad Gateway response now. This should (and used to) return 404 instead.

[databus23]

Hmm can't reproduce locally. I get the expected:

curl -i http://localhost:5100/api/v1/clusters/bla
HTTP/1.1 404 Not Found
Content-Type: application/json
Vary: Origin
Date: Wed, 20 Dec 2017 13:36:01 GMT
Content-Length: 35

{"code":404,"message":"Not found"}

[databus23]

ok I can reproduce in eu-nl-1

[databus23]

It seems the nginx-ingress is trying to redirect 404 to the default backend (which it shouldn't) and that fails with

2017/12/20 13:41:55 [error] 3302#3302: *116611 upstream prematurely closed connection while reading response header from upstream, client: 198.18.0.76, server: [REDACTED], request: "GET /api/clusters HTTP/2.0", upstream: "http://192.168.0.73:8080/", host: " [REDACTED]"

That shouldn't fail either (192.168.0.73 is the default backend)

[databus23]

the default backend is failing with:

2017/12/20 13:41:55 http: panic serving 192.168.0.72:36388: runtime error: index out of range
goroutine 104487 [running]:
net/http.(*conn).serve.func1(0xc42008a8c0)
	/usr/local/opt/go/libexec/src/net/http/server.go:1697 +0xd0
panic(0x6d4520, 0x8d3ae0)
	/usr/local/opt/go/libexec/src/runtime/panic.go:491 +0x283
main.errorHandler.func1(0x8a9660, 0xc4201182a0, 0xc420166300)
	/Users/d064935/.go/src/k8s.io/ingress-nginx/images/custom-error-pages/main.go:69 +0xeba
net/http.HandlerFunc.ServeHTTP(0xc42011a0e0, 0x8a9660, 0xc4201182a0, 0xc420166300)
	/usr/local/opt/go/libexec/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0x8de700, 0x8a9660, 0xc4201182a0, 0xc420166300)
	/usr/local/opt/go/libexec/src/net/http/server.go:2254 +0x130
net/http.serverHandler.ServeHTTP(0xc4201260d0, 0x8a9660, 0xc4201182a0, 0xc420166300)
	/usr/local/opt/go/libexec/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc42008a8c0, 0x8a9d60, 0xc4201ae6c0)
	/usr/local/opt/go/libexec/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
	/usr/local/opt/go/libexec/src/net/http/server.go:2720 +0x288

[databus23]

Looping in @auhlig

[auhlig]

also in the backend unexpected error opening file: open /www/4xxhtml: no such file or directory, which should actually open /www/4xx.html -.-

[auhlig]

It's here

[auhlig]

I cannot kubectl auth init|refresh in eu-nl-1 right now, but staging, na-us-1 looks good.

[databus23]

This has been fixed by deploying #134