[Actions] Add action group for notifying users of alert execution failure

[spong]

This is an enhancement request coming from user feedback for the option to be notified on alert/rule failure so they can immediately investigate what went wrong and resolve the issue.

Granularity of configuration wasn't specified, however it would make sense to support both a single onError action configured for all rules, or additional onError actions applied to individual alerts/rules (e.g. a rule writer may only want to notify themself via slack when their specific rule fails).

cc @pmuellr @marrasherrier

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

My immediate thought on this was that it probably makes sense to have an 'error' action group "built-in" like we now have "resolved". But of course, that would be painful to have to add that to every rule/alert. A "global" one would make more sense. But "global" is a problem, as in RBAC issues. Even a "space-specific" one seems like would have RBAC issues.

We've certainly mentioned things like this as part of "meta-alerting" - #49410 - and maybe this should also involve the new health bits - #79056

Which then maybe makes this feel like maybe a new alert, but maybe you'd need admin access to use. Perhaps the alert could just be to notify that there are problems, without indicating what they are specifically - just a link to a page that would provide relevant info given their roles.

[spong]

Which then maybe makes this feel like maybe a new alert, but maybe you'd need admin access to use. Perhaps the alert could just be to notify that there are problems, without indicating what they are specifically - just a link to a page that would provide relevant info given their roles.

That seems reasonable to me. I imagine most use cases will either be an individual user wanting to keep tabs on rules/alerts they created, or a small set of admin/manager users needing to know of any failure, in which case a link to a page with details should be sufficient for their needs.

[pmuellr]

Those use cases ^^^ sound right to me.

[darnautov]

This definitely would be useful for the Anomaly detection alert type. Many things might happen after the alert has been created, e.g. datafeed has been stopped or anomaly detection job has been deleted. Throwing an error and showing it in the Alert and Action UI doesn't suffice because depending on the significance of this alert, the user should take actions to resolve it, hence receiving a notification is critical.
We can create a special action group for this case (see #93009) but seems like having a notification about the Error state makes more sense and better UX in general.

[pmuellr]

@darnautov Another thing you might want to look into (if you haven't already), is alert navigation. This provides the ability for an alert type to provide link backs to another Kibana application, from the centralized alerting UIs.

Some doc here: https://github.com/elastic/kibana/blob/master/x-pack/plugins/alerts/README.md#alert-navigation

If you run Kibana with --run-examples, there's an example alert - people in space - which uses the alert navigation; source for that here: https://github.com/elastic/kibana/tree/master/x-pack/examples/alerting_example

I'm thinking this would be useful in the error cases, when a user ends up in the alerts UI, at least they will have a link into your app, without having to separately find and bring up that UI.

[mikecote]

Closing in favour of #49410.