Missing Kubernetes Events from Flagger

[tnishida1]

Expected Behavior

As the Canary Deployment rolls out, a corresponding Kubernetes Event should be created in the namespace of the application being deployed for each line in Flagger.

Current Behavior

As is shown by the logs below, and the following image, although the Canary Deployment had successfully been promoted and the logging had been reported, not all of the Kubernetes Events are being reported in the Flagger.

Context (Environment)

• Flagger Version: 0.20.4
• Istio Version: 1.2.2
• Kubernetes Version: 1.13
• EKS Version: 5.1.0
  We are utilizing flagger by rolling out a Canary Deployment of a microservice which is part of a larger monolith. We are utilizing a Slack Operator which watches at the namespace level on the cluster for any Kubernetes Events with an involvedObject.kind=Canary, and reports the information of the rollout back into Slack.

[stefanprodan]

Kubernetes does event compaction and I'm guessing that the success rollout message (since it doesn't change) it's just omitted. If you use a new canary object is the success rollout event pushed to Slack?

PS. Flagger can send events to Slack also https://docs.flagger.app/usage/alerting

[mrparkers]

Yes, it looks like Kubernetes is compacting duplicate events. Restarting the flagger pod seems to temporarily fix it, as well as using a new Canary object.

We do our own custom alerting around Canaries so I am not sure that the built-in Slack alerting will work for us. Would you accept a PR that adds an optional generic event webhook that flagger can use to send event payloads to a web server?

[stefanprodan]

There are a couple of webhook types that you could use, here are the docs https://docs.flagger.app/how-it-works#webhooks

[mrparkers]

Please correct me if I'm wrong, but my understanding of those webhooks is that flagger uses them as part of the canary analysis to determine whether or not it should proceed to the next step of the canary rollout.

What I'm looking for is a webhook that flagger can use to dump information about what it's doing (similar to how Kubernetes events are being used), so we can parse them and alert on these actions in a custom way. Is there a way that I can use the existing webhooks for this purpose?

[stefanprodan]

The pre rollout hook tells you that a canary has started and the post rollout hook tells you if it failed or not. Flagger posts a json object with the canary name, namespace and status, does this cover your use case?

[mrparkers]

I appreciate the suggestion, but we really value the feedback about the canary analysis progress that the Kubernetes events gave us, so we're looking to something that provides us the same or similar information.

[stefanprodan]

@mrparkers we could introduce a webhook of type event and post the json log to the specified endpoint.

[mrparkers]

@stefanprodan opened #401