Canary ingress nginx prevent update of primary ingress due to admission webhook

[vorozhko]

Hi all,

we have a problem with ingress admission webhook.
Using podinfo as example we did a canary deployment.

Flagger created second ingress and after rollout was done it switch "canary" annotation from "true" to "false":

apiVersion:  networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx-v2
    nginx.ingress.kubernetes.io/canary: "false"

I added "test" annotation to main Ingress to trigger update:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: podinfo
  labels:
    app: podinfo
  annotations:
    kubernetes.io/ingress.class: "nginx-v2"
    test: "test"
...

Now when I try to apply main Ingress file I get admission webhook error:

Error from server (BadRequest): error when creating "podinfo.yaml": 
admission webhook "validate.nginx.ingress.kubernetes.io" 
denied the request: host "example.com" and 
path "/" is already defined in ingress develop/podinfo-canary

podinfo Ingress

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: podinfo
  labels:
    app: podinfo
  annotations:
    kubernetes.io/ingress.class: "nginx-v2"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              serviceName: podinfo
              servicePort: 80
  tls:
  - hosts:
    - example.com
    secretName: example.com.wildcard

Flagger version: 1.6.1
Ingress nginx version: 0.43

[vorozhko]

The problem is in settings canary annotation to false

flagger/pkg/router/ingress.go

Line 221 in b8625d5

res[i.GetAnnotationWithPrefix("canary")] = "false"

That makes primary ingress and canary ingress identical.
So admission webhook "validate.nginx.ingress.kubernetes.io" don't allow any modification to ingress.

Alternative approaches could be:

• set canary weight to 0
• delete canary ingress

[stefanprodan]

Hmm setting canary weight to 0 is not going to stop header based routing. I really don't understand why the validation fails, what's the point of having an on/off switch if it doesn't work? This seems like a bug in the ingress validation webhook.

[vorozhko]

Agree I opened a ticket to ingess-nginx.

[vorozhko]

@stefanprodan nginx guys said canary:false in fact disable canary, so host/path conflict error is correct.
Can flagger delete the canary ingress instead?

[saitama-gh]

Hi all, same issue here.
Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "saitama-service" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "saitama-service.stg.peja.co" and path "/" is already defined in ingress apps/saitama-service-canary: cannot patch "saitama-service" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "saitama-service.stg.peja.co" and path "/" is already defined in ingress apps/saitama-service-canary helm.go:81: [debug] cannot patch "saitama-service" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "saitama-service.stg.peja.co" and path "/" is already defined in ingress apps/saitama-service-canary helm.sh/helm/v3/pkg/kube.(*Client).Update

[henriqcabral]

I have the same error, to solve I deleted the ValidatingWebhookConfiguration

[vorozhko]

@stefanprodan why don't delete Canary Ingress instead of setting it to 'false' if that solve the issue?

[vorozhko]

@stefanprodan is there workaround for this issue?
I am thinking to use kubectl command as post-rollout webhook to clean up the canary ingress from cluster. But, I am not sure what the best way to do that.

[stefanprodan]

The only workaround is to delete the nginx webhook.

[vorozhko]

@stefanprodan what about to keep canary:true, but set canary-weight: "0"?
That will not conflict with validation webhook.

[stefanprodan]

@vorozhko Would that work if there are headers/cookies conditions?

[vorozhko]

It wouldn't, but why not to drop all conditions at the end of the canary and set weight to "0"?

[stefanprodan]

@vorozhko yes that seems like a good solution, would you like to help out implementing this change?

[usrlocalang]

What is the solution for this issue? Setting canary-weight to zero?

[stefanprodan]

@usrlocalang you need to update Flagger to the last release that contains the fix for this issue.