Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

4.5.0 upgrade caused one of my ingresses to start returning 404s #9623

Closed
jeffcasavant opened this issue Feb 14, 2023 · 2 comments
Closed

4.5.0 upgrade caused one of my ingresses to start returning 404s #9623

jeffcasavant opened this issue Feb 14, 2023 · 2 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@jeffcasavant
Copy link

What happened:

I upgraded from Helm Chart 4.4.2 to 4.5.0 this evening and got 404s from NginX instead of responses on one of my services. I have an automated monitor set up to attempt to reach a URL every minute and it reported the failure. I downgraded to 4.4.2 and the 404 went away. I upgraded again and the 404s came back.

What you expected to happen:

The new version did not properly translate my ingress configuration into its reverse proxy configuration.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.6.3
  Build:         7ae9ca2f1d04ccdc818fd0d3fb0193266b739d68
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: DigitalOcean kubernetes service 1.25.4.do.0
  • Basic cluster related info:
kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP       OS-IMAGE                         KERNEL-VERSION           CONTAINER-RUNTIME
main-mgg8j   Ready    <none>   64d   v1.25.4   10.136.1.185   147.182.218.227   Debian GNU/Linux 11 (bullseye)   5.18.0-0.deb11.4-amd64   containerd://1.4.13
main-mgg8o   Ready    <none>   64d   v1.25.4   10.136.1.182   165.22.15.239     Debian GNU/Linux 11 (bullseye)   5.18.0-0.deb11.4-amd64   containerd://1.4.13
  • How was the ingress-nginx-controller installed:
└──> helm ls -A | grep -i ingress
ingress-nginx           ingress         21              2023-02-13 23:25:18.679940122 -0500 EST deployed        ingress-nginx-4.5.0     1.6.3
└──> helm -n ingress get values ingress-nginx
USER-SUPPLIED VALUES:
controller:
  kind: DaemonSet
  • Current State of the controller:
└──> kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.6.3
              helm.sh/chart=ingress-nginx-4.5.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress
Controller:   k8s.io/ingress-nginx
Events:       <none>
└──> kubectl -n ingress get all -A -o wide
NAMESPACE     NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE    CONTAINERS                                IMAGES                                                                                                                    SELECTOR
ingress       daemonset.apps/ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   159d   controller                                registry.k8s.io/ingress-nginx/controller:v1.6.3@sha256:b92667e0afde1103b736e6a3f00dd75ae66eec4e71827d19f19f471699e909d2   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-ngi
nx,app.kubernetes.io/name=ingress-nginx
kube-system   daemonset.apps/cilium                     2         2         2       2            2           <none>                   420d   cilium-agent                              quay.io/cilium/cilium:v1.11.11@sha256:fad1627d1c52ff084c3147bbeba7a7b64c60562db38c5e7a9b37c710a8f1d3c1                    k8s-app=cilium,kubernetes.io/cluster-service=true
kube-system   daemonset.apps/cpc-bridge-proxy           2         2         2       2            2           <none>                   64d    cpc-bridge-proxy                          digitalocean/cpbridge:1.21.6                                                                                              app=cpc-bridge-proxy
kube-system   daemonset.apps/csi-do-node                2         2         2       2            2           <none>                   420d   csi-node-driver-registrar,csi-do-plugin   registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.0,docker.io/digitalocean/do-csi-plugin:v4.4.1                  app=csi-do-node
kube-system   daemonset.apps/do-node-agent              2         2         2       2            2           kubernetes.io/os=linux   420d   do-node-agent                             docker.io/digitalocean/do-agent:3.11.0                                                                                    app=do-node-agent
kube-system   daemonset.apps/konnectivity-agent         2         2         2       2            2           <none>                   64d    konnectivity-agent                        registry.k8s.io/kas-network-proxy/proxy-agent:v0.0.35                                                                     k8s-app=konnectivity-agent
kube-system   daemonset.apps/kube-proxy                 2         2         2       2            2           <none>                   420d   kube-proxy                                registry.k8s.io/kube-proxy:v1.25.4                                                                                        k8s-app=kube-proxy,tier=node
└──> kubectl -n ingress describe po ingress-nginx-controller-qd49h
Name:             ingress-nginx-controller-qd49h
Namespace:        ingress
Priority:         0
Service Account:  ingress-nginx
Node:             main-mgg8o/10.136.1.182
Start Time:       Mon, 13 Feb 2023 23:26:48 -0500
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  controller-revision-hash=886dd8b84
                  pod-template-generation=12
Annotations:      <none>
Status:           Running
IP:               10.244.2.15
IPs:
  IP:           10.244.2.15
Controlled By:  DaemonSet/ingress-nginx-controller
Containers:
  controller:
    Container ID:  containerd://c91cd2d1a51fcb1873fd75002aebcf7e9d58b1f8f4e6bf1d944cb29568f035db
    Image:         registry.k8s.io/ingress-nginx/controller:v1.6.3@sha256:b92667e0afde1103b736e6a3f00dd75ae66eec4e71827d19f19f471699e909d2
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:b92667e0afde1103b736e6a3f00dd75ae66eec4e71827d19f19f471699e909d2
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Mon, 13 Feb 2023 23:26:49 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-qd49h (v1:metadata.name)
      POD_NAMESPACE:  ingress (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lgpkf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-lgpkf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason     Age    From                      Message
  ----    ------     ----   ----                      -------
  Normal  Scheduled  3m47s  default-scheduler         Successfully assigned ingress/ingress-nginx-controller-qd49h to main-mgg8o
  Normal  Pulled     3m47s  kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.6.3@sha256:b92667e0afde1103b736e6a3f00dd75ae66eec4e71827d19f19f471699e909d2" already present on machine
  Normal  Created    3m47s  kubelet                   Created container controller
  Normal  Started    3m46s  kubelet                   Started container controller
  Normal  RELOAD     3m43s  nginx-ingress-controller  NGINX reload triggered due to a change in configuration
  └──> kubectl -n ingress describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.6.3
                          helm.sh/chart=ingress-nginx-4.5.0
Annotations:              kubernetes.digitalocean.com/load-balancer-id: 079d5fc9-19d9-467d-a2b4-d8c2aee87693
                          meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: ingress
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.245.240.103
IPs:                      10.245.240.103
LoadBalancer Ingress:     137.184.242.12
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30010/TCP
Endpoints:                10.244.1.95:80,10.244.2.15:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30778/TCP
Endpoints:                10.244.1.95:443,10.244.2.15:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
  • Current state of ingress object, if applicable:
└──> kubectl -n synapse get all,ing -o wide
NAME                                           READY   STATUS    RESTARTS   AGE    IP             NODE         NOMINATED NODE   READINESS GATES
pod/synapse-6ffddd957b-bzmfr                   1/1     Running   0          6d6h   10.244.2.52    main-mgg8o   <none>           <none>
pod/synapse-background1-6c4d994c6-j97sb        1/1     Running   0          6d6h   10.244.1.41    main-mgg8j   <none>           <none>
pod/synapse-clientminutiae1-68b885f885-4wzc2   1/1     Running   0          6d6h   10.244.2.115   main-mgg8o   <none>           <none>
pod/synapse-fedsender1-7b547886d4-l7rn6        1/1     Running   0          6d6h   10.244.2.93    main-mgg8o   <none>           <none>
pod/synapse-generic1-7d6558bfc8-zcb2n          1/1     Running   0          6d6h   10.244.1.32    main-mgg8j   <none>           <none>
pod/synapse-generic2-56768768b-4j8rz           1/1     Running   0          6d6h   10.244.1.115   main-mgg8j   <none>           <none>

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE    SELECTOR
service/web                   ClusterIP   10.245.40.218    <none>        8008/TCP,8083/TCP,8090/TCP,9093/TCP,9094/TCP   198d   app=synapse
service/web-clientminutiae1   ClusterIP   10.245.153.169   <none>        8090/TCP,8091/TCP                              176d   app=synapse-clientminutiae1
service/web-generic           ClusterIP   10.245.102.115   <none>        8083/TCP                                       176d   app=synapse-generic
service/web-synapse           ClusterIP   10.245.230.100   <none>        8008/TCP,9093/TCP,9094/TCP                     176d   app=synapse

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS        IMAGES                         SELECTOR
deployment.apps/synapse                   1/1     1            1           155d   synapse           matrixdotorg/synapse:v1.76.0   app=synapse
deployment.apps/synapse-background1       1/1     1            1           155d   background1       matrixdotorg/synapse:v1.76.0   app=synapse-background1
deployment.apps/synapse-clientminutiae1   1/1     1            1           155d   clientminutiae1   matrixdotorg/synapse:v1.76.0   app=synapse-clientminutiae1
deployment.apps/synapse-fedsender1        1/1     1            1           155d   fedsender1        matrixdotorg/synapse:v1.76.0   app=synapse-fedsender1
deployment.apps/synapse-generic1          1/1     1            1           155d   generic1          matrixdotorg/synapse:v1.76.0   app=synapse-generic
deployment.apps/synapse-generic2          1/1     1            1           155d   generic2          matrixdotorg/synapse:v1.76.0   app=synapse-generic

NAME                                                 DESIRED   CURRENT   READY   AGE    CONTAINERS        IMAGES                         SELECTOR
replicaset.apps/synapse-5f99d57758                   0         0         0       131d   synapse           matrixdotorg/synapse:v1.68.0   app=synapse,pod-template-hash=5f99d57758
replicaset.apps/synapse-688bcff75c                   0         0         0       83d    synapse           matrixdotorg/synapse:v1.72.0   app=synapse,pod-template-hash=688bcff75c
replicaset.apps/synapse-69fdcfb969                   0         0         0       119d   synapse           matrixdotorg/synapse:v1.69.0   app=synapse,pod-template-hash=69fdcfb969
replicaset.apps/synapse-6ffddd957b                   1         1         1       12d    synapse           matrixdotorg/synapse:v1.76.0   app=synapse,pod-template-hash=6ffddd957b
replicaset.apps/synapse-775dbc8878                   0         0         0       108d   synapse           matrixdotorg/synapse:v1.70.1   app=synapse,pod-template-hash=775dbc8878
replicaset.apps/synapse-7869696747                   0         0         0       110d   synapse           matrixdotorg/synapse:v1.70.0   app=synapse,pod-template-hash=7869696747
replicaset.apps/synapse-788d9ccf46                   0         0         0       97d    synapse           matrixdotorg/synapse:v1.71.0   app=synapse,pod-template-hash=788d9ccf46
replicaset.apps/synapse-798dbd8bd                    0         0         0       69d    synapse           matrixdotorg/synapse:v1.73.0   app=synapse,pod-template-hash=798dbd8bd
replicaset.apps/synapse-7d4694f55d                   0         0         0       55d    synapse           matrixdotorg/synapse:v1.74.0   app=synapse,pod-template-hash=7d4694f55d
replicaset.apps/synapse-background1-595fb58f8f       0         0         0       83d    background1       matrixdotorg/synapse:v1.72.0   app=synapse-background1,pod-template-hash=595fb58f8f
replicaset.apps/synapse-background1-5f5cc6fcdb       0         0         0       55d    background1       matrixdotorg/synapse:v1.74.0   app=synapse-background1,pod-template-hash=5f5cc6fcdb
replicaset.apps/synapse-background1-6c4d994c6        1         1         1       12d    background1       matrixdotorg/synapse:v1.76.0   app=synapse-background1,pod-template-hash=6c4d994c6
replicaset.apps/synapse-background1-6f7d67cbb4       0         0         0       139d   background1       matrixdotorg/synapse:v1.68.0   app=synapse-background1,pod-template-hash=6f7d67cbb4
replicaset.apps/synapse-background1-768b78ff75       0         0         0       110d   background1       matrixdotorg/synapse:v1.70.0   app=synapse-background1,pod-template-hash=768b78ff75
replicaset.apps/synapse-background1-7c5f7998f        0         0         0       27d    background1       matrixdotorg/synapse:v1.75.0   app=synapse-background1,pod-template-hash=7c5f7998f
replicaset.apps/synapse-background1-7ccfb9fbdb       0         0         0       97d    background1       matrixdotorg/synapse:v1.71.0   app=synapse-background1,pod-template-hash=7ccfb9fbdb
replicaset.apps/synapse-background1-7d5764d797       0         0         0       119d   background1       matrixdotorg/synapse:v1.69.0   app=synapse-background1,pod-template-hash=7d5764d797
replicaset.apps/synapse-background1-7d6db76db8       0         0         0       69d    background1       matrixdotorg/synapse:v1.73.0   app=synapse-background1,pod-template-hash=7d6db76db8
replicaset.apps/synapse-background1-8c56869b6        0         0         0       108d   background1       matrixdotorg/synapse:v1.70.1   app=synapse-background1,pod-template-hash=8c56869b6
replicaset.apps/synapse-background1-b78cfdfc4        0         0         0       131d   background1       matrixdotorg/synapse:v1.68.0   app=synapse-background1,pod-template-hash=b78cfdfc4
replicaset.apps/synapse-clientminutiae1-549b49ddc    0         0         0       97d    clientminutiae1   matrixdotorg/synapse:v1.71.0   app=synapse-clientminutiae1,pod-template-hash=549b49ddc
replicaset.apps/synapse-clientminutiae1-5598d64f7c   0         0         0       139d   clientminutiae1   matrixdotorg/synapse:v1.68.0   app=synapse-clientminutiae1,pod-template-hash=5598d64f7c
replicaset.apps/synapse-clientminutiae1-6476866b76   0         0         0       83d    clientminutiae1   matrixdotorg/synapse:v1.72.0   app=synapse-clientminutiae1,pod-template-hash=6476866b76
replicaset.apps/synapse-clientminutiae1-65f4577ff6   0         0         0       27d    clientminutiae1   matrixdotorg/synapse:v1.75.0   app=synapse-clientminutiae1,pod-template-hash=65f4577ff6
replicaset.apps/synapse-clientminutiae1-66694fbcc4   0         0         0       110d   clientminutiae1   matrixdotorg/synapse:v1.70.0   app=synapse-clientminutiae1,pod-template-hash=66694fbcc4
replicaset.apps/synapse-clientminutiae1-68b885f885   1         1         1       12d    clientminutiae1   matrixdotorg/synapse:v1.76.0   app=synapse-clientminutiae1,pod-template-hash=68b885f885
replicaset.apps/synapse-clientminutiae1-6cd67754c9   0         0         0       55d    clientminutiae1   matrixdotorg/synapse:v1.74.0   app=synapse-clientminutiae1,pod-template-hash=6cd67754c9
replicaset.apps/synapse-clientminutiae1-6cf48fb54d   0         0         0       119d   clientminutiae1   matrixdotorg/synapse:v1.69.0   app=synapse-clientminutiae1,pod-template-hash=6cf48fb54d
replicaset.apps/synapse-clientminutiae1-6dc66dcf9c   0         0         0       69d    clientminutiae1   matrixdotorg/synapse:v1.73.0   app=synapse-clientminutiae1,pod-template-hash=6dc66dcf9c
replicaset.apps/synapse-clientminutiae1-85b5dfb975   0         0         0       108d   clientminutiae1   matrixdotorg/synapse:v1.70.1   app=synapse-clientminutiae1,pod-template-hash=85b5dfb975
replicaset.apps/synapse-clientminutiae1-9ff746c99    0         0         0       131d   clientminutiae1   matrixdotorg/synapse:v1.68.0   app=synapse-clientminutiae1,pod-template-hash=9ff746c99
replicaset.apps/synapse-dcbdb9785                    0         0         0       27d    synapse           matrixdotorg/synapse:v1.75.0   app=synapse,pod-template-hash=dcbdb9785
replicaset.apps/synapse-fedsender1-565b9bbdf9        0         0         0       110d   fedsender1        matrixdotorg/synapse:v1.70.0   app=synapse-fedsender1,pod-template-hash=565b9bbdf9
replicaset.apps/synapse-fedsender1-5768c4955d        0         0         0       55d    fedsender1        matrixdotorg/synapse:v1.74.0   app=synapse-fedsender1,pod-template-hash=5768c4955d
replicaset.apps/synapse-fedsender1-665fdb9cf9        0         0         0       27d    fedsender1        matrixdotorg/synapse:v1.75.0   app=synapse-fedsender1,pod-template-hash=665fdb9cf9
replicaset.apps/synapse-fedsender1-666657c858        0         0         0       69d    fedsender1        matrixdotorg/synapse:v1.73.0   app=synapse-fedsender1,pod-template-hash=666657c858
replicaset.apps/synapse-fedsender1-6c5b788c6         0         0         0       131d   fedsender1        matrixdotorg/synapse:v1.68.0   app=synapse-fedsender1,pod-template-hash=6c5b788c6
replicaset.apps/synapse-fedsender1-6cf646d685        0         0         0       83d    fedsender1        matrixdotorg/synapse:v1.72.0   app=synapse-fedsender1,pod-template-hash=6cf646d685
replicaset.apps/synapse-fedsender1-77474d7d94        0         0         0       139d   fedsender1        matrixdotorg/synapse:v1.68.0   app=synapse-fedsender1,pod-template-hash=77474d7d94
replicaset.apps/synapse-fedsender1-7b547886d4        1         1         1       12d    fedsender1        matrixdotorg/synapse:v1.76.0   app=synapse-fedsender1,pod-template-hash=7b547886d4
replicaset.apps/synapse-fedsender1-8685479c7         0         0         0       119d   fedsender1        matrixdotorg/synapse:v1.69.0   app=synapse-fedsender1,pod-template-hash=8685479c7
replicaset.apps/synapse-fedsender1-9dd86dbcb         0         0         0       97d    fedsender1        matrixdotorg/synapse:v1.71.0   app=synapse-fedsender1,pod-template-hash=9dd86dbcb
replicaset.apps/synapse-fedsender1-ccdd698d4         0         0         0       108d   fedsender1        matrixdotorg/synapse:v1.70.1   app=synapse-fedsender1,pod-template-hash=ccdd698d4
replicaset.apps/synapse-ffd8c5464                    0         0         0       139d   synapse           matrixdotorg/synapse:v1.68.0   app=synapse,pod-template-hash=ffd8c5464
replicaset.apps/synapse-generic1-65b84bf95d          0         0         0       97d    generic1          matrixdotorg/synapse:v1.71.0   app=synapse-generic,pod-template-hash=65b84bf95d
replicaset.apps/synapse-generic1-66486dc47b          0         0         0       69d    generic1          matrixdotorg/synapse:v1.73.0   app=synapse-generic,pod-template-hash=66486dc47b
replicaset.apps/synapse-generic1-6f6d9f99d           0         0         0       131d   generic1          matrixdotorg/synapse:v1.68.0   app=synapse-generic,pod-template-hash=6f6d9f99d
replicaset.apps/synapse-generic1-6fb8fd5c6c          0         0         0       83d    generic1          matrixdotorg/synapse:v1.72.0   app=synapse-generic,pod-template-hash=6fb8fd5c6c
replicaset.apps/synapse-generic1-758dfcf49c          0         0         0       110d   generic1          matrixdotorg/synapse:v1.70.0   app=synapse-generic,pod-template-hash=758dfcf49c
replicaset.apps/synapse-generic1-7cfd8499c6          0         0         0       139d   generic1          matrixdotorg/synapse:v1.68.0   app=synapse-generic,pod-template-hash=7cfd8499c6
replicaset.apps/synapse-generic1-7d6558bfc8          1         1         1       12d    generic1          matrixdotorg/synapse:v1.76.0   app=synapse-generic,pod-template-hash=7d6558bfc8
replicaset.apps/synapse-generic1-986b675c9           0         0         0       119d   generic1          matrixdotorg/synapse:v1.69.0   app=synapse-generic,pod-template-hash=986b675c9
replicaset.apps/synapse-generic1-b647d74d4           0         0         0       55d    generic1          matrixdotorg/synapse:v1.74.0   app=synapse-generic,pod-template-hash=b647d74d4
replicaset.apps/synapse-generic1-bdb44fdbf           0         0         0       27d    generic1          matrixdotorg/synapse:v1.75.0   app=synapse-generic,pod-template-hash=bdb44fdbf
replicaset.apps/synapse-generic1-f66ff95b6           0         0         0       108d   generic1          matrixdotorg/synapse:v1.70.1   app=synapse-generic,pod-template-hash=f66ff95b6
replicaset.apps/synapse-generic2-557677f64f          0         0         0       27d    generic2          matrixdotorg/synapse:v1.75.0   app=synapse-generic,pod-template-hash=557677f64f
replicaset.apps/synapse-generic2-55f6f4c4b8          0         0         0       139d   generic2          matrixdotorg/synapse:v1.68.0   app=synapse-generic,pod-template-hash=55f6f4c4b8
replicaset.apps/synapse-generic2-56768768b           1         1         1       12d    generic2          matrixdotorg/synapse:v1.76.0   app=synapse-generic,pod-template-hash=56768768b
replicaset.apps/synapse-generic2-5958f4487b          0         0         0       69d    generic2          matrixdotorg/synapse:v1.73.0   app=synapse-generic,pod-template-hash=5958f4487b
replicaset.apps/synapse-generic2-59d78b48f4          0         0         0       97d    generic2          matrixdotorg/synapse:v1.71.0   app=synapse-generic,pod-template-hash=59d78b48f4
replicaset.apps/synapse-generic2-5fd9d9b7fd          0         0         0       108d   generic2          matrixdotorg/synapse:v1.70.1   app=synapse-generic,pod-template-hash=5fd9d9b7fd
replicaset.apps/synapse-generic2-6746f78c9b          0         0         0       55d    generic2          matrixdotorg/synapse:v1.74.0   app=synapse-generic,pod-template-hash=6746f78c9b
replicaset.apps/synapse-generic2-6d86558545          0         0         0       110d   generic2          matrixdotorg/synapse:v1.70.0   app=synapse-generic,pod-template-hash=6d86558545
replicaset.apps/synapse-generic2-746c69676c          0         0         0       83d    generic2          matrixdotorg/synapse:v1.72.0   app=synapse-generic,pod-template-hash=746c69676c
replicaset.apps/synapse-generic2-7cf66586ff          0         0         0       119d   generic2          matrixdotorg/synapse:v1.69.0   app=synapse-generic,pod-template-hash=7cf66586ff
replicaset.apps/synapse-generic2-7fc5bfbb6b          0         0         0       131d   generic2          matrixdotorg/synapse:v1.68.0   app=synapse-generic,pod-template-hash=7fc5bfbb6b

NAME                                CLASS    HOSTS                 ADDRESS          PORTS     AGE
ingress.networking.k8s.io/synapse   <none>   matrix.casavant.org   137.184.242.12   80, 443   198d
└──> kubectl -n synapse describe ing synapse
Name:             synapse
Labels:           <none>
Namespace:        synapse
Address:          137.184.242.12
Ingress Class:    <none>
Default backend:  <default>
TLS:
  synapse terminates matrix.casavant.org
Rules:
  Host                 Path  Backends
  ----                 ----  --------
  matrix.casavant.org
                       /                                                                    web-synapse:8008 (10.244.2.52:8008)
                       /_matrix/client/(r0|v3)/sync$                                        web-generic:8083 (10.244.1.115:8083,10.244.1.32:8083)
                       /_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(typing|presence)   web-clientminutiae1:8090 (10.244.2.115:8090)
                       /_matrix/client/(r0|v3|unstable)/rooms/.*/(receipt|read_markers)     web-clientminutiae1:8090 (10.244.2.115:8090)
Annotations:           cert-manager.io/cluster-issuer: letsencrypt-prod
                       kubernetes.io/ingress.class: nginx
                       nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
                       nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
                       nginx.ingress.kubernetes.io/use-regex: true
Events:
  Type    Reason             Age                From                       Message
  ----    ------             ----               ----                       -------
  Normal  Sync               44m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               42m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               35m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               34m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               32m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               31m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               27m                nginx-ingress-controller   Scheduled for sync
  Normal  UpdateCertificate  21m                cert-manager-ingress-shim  Successfully updated Certificate "synapse"
  Normal  Sync               20m (x3 over 25m)  nginx-ingress-controller   Scheduled for sync
  Normal  Sync               20m (x3 over 24m)  nginx-ingress-controller   Scheduled for sync
  Normal  Sync               18m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               16m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               15m                nginx-ingress-controller   Scheduled for sync
  Normal  Sync               6m30s              nginx-ingress-controller   Scheduled for sync
  Normal  Sync               5m29s              nginx-ingress-controller   Scheduled for sync
  └──> curl -v https://matrix.casavant.org/_matrix/client/versions
*   Trying 137.184.242.12:443...
* Connected to matrix.casavant.org (137.184.242.12) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* [CONN-0-0][CF-SSL] TLSv1.0 (OUT), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Finished (20):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=matrix-media-repo.bridges.matrix.casavant.org
*  start date: Jan  8 23:19:47 2023 GMT
*  expire date: Apr  8 23:19:46 2023 GMT
*  subjectAltName: host "matrix.casavant.org" matched cert's "matrix.casavant.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: GET]
* h2h3 [:path: /_matrix/client/versions]
* h2h3 [:scheme: https]
* h2h3 [:authority: matrix.casavant.org]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x55d5d114fec0)
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /_matrix/client/versions HTTP/2
> Host: matrix.casavant.org
> user-agent: curl/7.87.0
> accept: */*
>
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Supplemental data (23):
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404
< date: Tue, 14 Feb 2023 04:32:54 GMT
< content-type: text/html
< content-length: 146
< strict-transport-security: max-age=15724800; includeSubDomains
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host matrix.casavant.org left intact

How to reproduce this issue:

Here's my ingress configuration that is breaking:

---
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: synapse
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: '3600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '3600'
    nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
  tls:
    - hosts:
        - matrix.casavant.org
      secretName: synapse
  rules:
    - host: matrix.casavant.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-synapse
                port:
                  number: 8008
          - path: /_matrix/client/(r0|v3)/sync$
            pathType: Exact
            backend:
              service:
                name: web-generic
                port:
                  number: 8083
          - path: /_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(typing|presence)
            pathType: Prefix
            backend:
              service:
                name: web-clientminutiae1
                port:
                  number: 8090
          - path: /_matrix/client/(r0|v3|unstable)/rooms/.*/(receipt|read_markers)
            pathType: Prefix
            backend:
              service:
                name: web-clientminutiae1
                port:
                  number: 8090

Anything else we need to know:
@jeffcasavant jeffcasavant added the kind/bug Categorizes issue or PR as related to a bug. label Feb 14, 2023
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Feb 14, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jeffcasavant
Copy link
Author

Closing duplicate of #9616

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
[SIG Network] Ingress NGINX
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jeffcasavant @k8s-ci-robot