[nginx] default cert not served in absence of servername (since beta.4+)

[donaldguy]

Presumably the result of #614, it is now the case that SNI must be used by a client to complete a TLS handshake even when the servername is ignored on the server side

While I first encountered in production with a valid star cert, I have successfully reproduced in minikube as follows:

$ minikube version
minikube version: v0.18.0
$ minikube start
Starting local Kubernetes cluster...
Starting VM...
Downloading Minikube ISO
 89.51 MB / 89.51 MB [==============================================] 100.00% 0s
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster.

$ minikube addons disable ingress
ingress was successfully disabled

$ minikube ip
192.168.99.100

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -subj '/CN=192.168.99.100' -nodes
Generating a 4096 bit RSA private key
..............................++
.......................................................................................................++
writing new private key to 'key.pem'
-----

$ kubectl config current-context
minikube

$ kubectl create secret tls selfsigned --key=key.pem --cert=cert.pem
secret "selfsigned" created

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/7033b70b578640baf255b84b1642bb91c112fa9b/deploy/addons/ingress/ingress-configmap.yaml
configmap "nginx-load-balancer-conf" configured

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/7033b70b578640baf255b84b1642bb91c112fa9b/deploy/addons/ingress/ingress-svc.yaml
service "default-http-backend" configured

$ wget --quiet https://raw.githubusercontent.com/kubernetes/minikube/7033b70b578640baf255b84b1642bb91c112fa9b/deploy/addons/ingress/ingress-rc.yaml
$ echo "        - --default-ssl-certificate=default/selfsigned" >> ingress-rc.yaml
$ kubectl apply -f ingress-rc.yaml
replicationcontroller "default-http-backend" configured
replicationcontroller "nginx-ingress-controller" configured

$ curl 192.168.99.100
default backend - 404

$ openssl s_client -connect 192.168.99.100:443
openssl s_client -connect 192.168.99.101:443
CONNECTED(00000003)
140736382415880:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 290 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1494263260
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

openssl s_client -connect 192.168.99.100:443 -servername asdjsdjksdkjasd
CONNECTED(00000003)
depth=0 CN = 192.168.99.100
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = 192.168.99.100
verify return:1
---
Certificate chain
 0 s:/CN=192.168.99.100
   i:/CN=192.168.99.100
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFLzCCAxegAwIBAgIJAIVz6t7lfJaDMA0GCSqGSIb3DQEBBQUAMBkxFzAVBgNV
BAMTDjE5Mi4xNjguOTkuMTAwMB4XDTE3MDUwODE1NTgyN1oXDTE4MDUwODE1NTgy
N1owGTEXMBUGA1UEAxMOMTkyLjE2OC45OS4xMDAwggIiMA0GCSqGSIb3DQEBAQUA
A4ICDwAwggIKAoICAQDBxMGXADCDAwb9Z3ptce2C0iCpSppyVQUCViDY1TV5XFvs
LAwbJOJmb0qelJlx1ihHJ6Osa1OMkkBY2dxZZtU4bhbWiw7ubOgKVR3LuBaKf7td
E5nE5u0FME6/23UfDoOVhycMeYHKeWnUvenZ0uRXquQ454giBSL/ysemYkKMYMxF
YLBHhSyTsM0do0DsqqXCIzJyzXbLUOI/xP4uIrUXm79i1zy43FgYE+1RUN57ngGT
1EwGInZX/P0yJ+LdhAfnReU0HpF8ZAXizxlHeWZJSTLcY2dnOxOgLecrBhtBqCLe
H2SMylObAEI1IO/Xd3LtvXWFufRVqSVVm0wJPkuQ/oyWRxz3e5KYtUL1dDlbgxLg
oiTksElA7nzFXqQeBQMHCn49JrJDS6u9qg0KYXNwoC287FQe4+7+OoyOjkj0o9/N
SWJD6QVQCH6zmf3oKxcz4iLhUoWumuXxxnFLQmugBl2tGvqME0uLQbO7S220yFpB
8dQDb2YMW7zYE8UocLdE73gTRKIeZFqvc+xMe9mMRSzPJeBRlaZdKLfSZ3EpKp3u
O4pb0mkv/ULQHtCMm1Ngcm5gp31znRvxRqErTRsLe149YPZ03l71nrRTlhiMxGA6
aD1A+/YQrJBpTwtTQizAvVB7Gu3XSl28X1IDPtQyPl99zZzgW0qw9P157U0dhQID
AQABo3oweDAdBgNVHQ4EFgQUQ6vyakO9alLAgXWIwy9wF0Ce/9MwSQYDVR0jBEIw
QIAUQ6vyakO9alLAgXWIwy9wF0Ce/9OhHaQbMBkxFzAVBgNVBAMTDjE5Mi4xNjgu
OTkuMTAwggkAhXPq3uV8loMwDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOC
AgEALL3F7z8oqUoo98Qz0unFSvLJ7RKGenRTBE6LqJvtg9oZ9D+hdkgYQkj/4BhJ
fAJhtFTfXfKhmO006dh5RllHpkEruvwZcTgfoxG6CU5EfmqLKDw46QhBdh1xPqnB
6mxJjq92lFVM8LzqsflZ9PoU0C/ExUBLGJciT5pXaXa7jxwmS+mPunC2imtoMJq4
Cn333tv61YHFdDuXBIEhvTi3yrSim3xV2S6MxkPMrmtiq0/3QfOXnwAFtX1R1yX9
9c7sCQ8G246fTUL1vGXNUeCQxhz6DSwi+NVKpevDDKwyn/DpgHqyQ6DnUsTNSZKm
lllKh+nJmnGr5vlaNR57mb9ZrYvXBXy4IGFjp1UBiHhpyiD/Y/wNVTHBlz7PQnkT
uii0LmBYGBG94HF7senJNXD76ICkni8sKq6dC4N31CLnk0Ue8h7jYn8LagwUDkQA
b7OBXU7Yo+lOGHwBSRwBaK7yhzLNZQbEDI7Hk4z1xs5TAr2Xc6LadIGE5+nhALQ+
xFN57qCWzRbtup78ATSkh1tF/MviylEqkNCPJDSJdTsOk7zX/pzRQT5ZwCWE+7iW
+eJ0ICToMS5666SgijQkW8tqaURoWFGmtGl+GPewov89gvni7987UUdA+d4ikiKF
HrKpJM+Hw0gc2G2IRA73OlcMIFnm4CYVm8xbkedhmua5JMw=
-----END CERTIFICATE-----
subject=/CN=192.168.99.100
issuer=/CN=192.168.99.100
---
No client certificate CA names sent
---
SSL handshake has read 2310 bytes and written 463 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: ED633B9AE7B0812773062A544E3BD8FB390419876958D40ED660FA214CA31C50
    Session-ID-ctx:
    Master-Key: 457C31C45221BA1250CC16AD90B522CEC5F948E0D9528245099ACE888ADDBCE50B9F5E329F20E9A9B65CB4EF09CE5EB5
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 95 99 cd fa c0 43 ac 05-21 82 9f d9 37 a9 5f 8f   .....C..!...7._.
    0010 - 0a f7 95 31 fa 1d fe 65-3e 14 ae 0e 97 d1 12 36   ...1...e>......6
    0020 - b1 fb 24 f8 1d 3b a9 15-03 ef e4 6b e6 18 f4 77   ..$..;.....k...w
    0030 - 8a bf c3 08 d3 f9 8d b0-11 9b 4d 6f 9b d2 2c ec   ..........Mo..,.
    0040 - f6 ba 8b c6 53 4d e0 f3-dd dd e7 e6 61 7b a2 71   ....SM......a{.q
    0050 - 25 91 d5 b8 89 f1 c6 da-3a 0b c6 26 c6 19 68 3d   %.......:..&..h=
    0060 - a1 41 d9 20 fc 1a dc 3e-54 ee cf 40 e8 97 41 78   .A. ...>[email protected]
    0070 - d5 f7 79 a6 50 b9 09 bd-04 62 50 f2 4e b3 a5 f4   ..y.P....bP.N...
    0080 - eb 59 89 77 b4 49 b7 35-6f 06 03 78 34 40 da ae   .Y.w.I.5o..x4@..
    0090 - 3a 9f 86 3b 2b 7f ef 1e-11 93 10 da c5 99 0a a4   :..;+...........
    00a0 - 2b 09 a3 bd 86 ba b6 a1-50 d2 c2 20 02 11 03 70   +.......P.. ...p
    00b0 - 8b 0f df d8 5d 91 c1 8f-6e 69 da 05 46 81 fe c5   ....]...ni..F...

    Start Time: 1494263289
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

I have verified this happens in production with both beta 4 and beta 5. Reverting to beta 3 seems to work.

[rikatz]

@donaldguy Can you please cat the /etc/nginx/nginx.conf generated by the ingress-controller and send here or in a gist?

Thanks.

[donaldguy]

It's pretty much what you'd expect:

https://gist.github.com/donaldguy/d56fa2a5d16195cec840f9e80ec7a198

[rikatz]

Yup, it seems there is no 443 port listening in this.

@aledbf Don't know were this happens, maybe in this SSL Passthrough refactor.

[donaldguy]

additionally I can confirm that

diff -u <(cat cert.pem key.pem) <(kubectl -n kube-system exec nginx-ingress-controller-b56td cat /ingress-controller/ssl/default-selfsigned.pem)                                                                        9706  ☸ minikube  I
--- /dev/fd/11	2017-05-08 17:14:17.000000000 -0400
+++ /dev/fd/13	2017-05-08 17:14:17.000000000 -0400
@@ -28,6 +28,7 @@
 +eJ0ICToMS5666SgijQkW8tqaURoWFGmtGl+GPewov89gvni7987UUdA+d4ikiKF
 HrKpJM+Hw0gc2G2IRA73OlcMIFnm4CYVm8xbkedhmua5JMw=
 -----END CERTIFICATE-----
+
 -----BEGIN RSA PRIVATE KEY-----
 MIIJKAIBAAKCAgEAwcTBlwAwgwMG/Wd6bXHtgtIgqUqaclUFAlYg2NU1eVxb7CwM
 GyTiZm9KnpSZcdYoRyejrGtTjJJAWNncWWbVOG4W1osO7mzoClUdy7gWin+7XROZ```

[rikatz]

Yeap, it seems that in the config the 443 port is not being openned. Need to check this.

[donaldguy]

I'd put decent odds that the site of the bug is here: https://github.com/kubernetes/ingress/blame/nginx-0.9.0-beta.5/controllers/nginx/pkg/cmd/controller/tcp.go#L49-L60

[donaldguy]

In particular it would appear that p.Default is always nil

[rikatz]

Hum. Don't know when this was inserted in the code and why is there.

Anyway I use a modified nginx.tmpl that uses SSL directly (as my traffic is only HTTP/s).

But will check this ASAP to see what and when happens.

I suggest you doing a modification in this part of the code, to see if this solves. If that's the case, open a PR :)

Thanks!