Create metric for flux manifest errors

[mpashka]

Describe the feature
Add metric to inform users about invalid manifest or flux inability to apply manifest to k8s by some reason.

Sometimes user can make a mistake and push wrong manifest into git. Or sometimes manifest is correct but flux can't apply it - it is common use case for big configmaps - flux uses "apply" to push configmap to kubernetes and that implies saving configmap content into annotations. And in kubernetes there is limitation on annotations size. So if configmap size is bigger than some limit (something like 255Kb) then kubectl apply reports error and manifest can't be applied. There are flux events that can be used to detect this but in standalone flux installation it's much more convenient to create metrics with information about manifest errors. Metrics that can be easily checked any time. Also that metrics can be used to rise prometheus alarm which is de facto standard for k8s installations.

Expected behavior
I expect new metric with a number of invalid manifest files in flux git repository. And metric with a number of errors that occured during applying manifests to kubernetes. And possible a metric with a total number of manifests, but that one is only for reference.

Also it can be good idea to create flux prometheus dashboard that can be used for checking flux health and prometheus alarms for flux that can be used to inform users about invalid manifests.

[ksaritek]

@stefanprodan & @mpashka, is it a difficult ticket for a newbie? is anybody working on that?

[2opremio]

I think we can close this, since it was addressed by #2535

[ArthurSens]

Is it possible to add git-repository, git-path, and file to this metric?
If I'm using one Prometheus server to monitor multiple cluster/repositories, I'd like to create an alert that tells me exactly where is the error that is causing the failed sync