[SIEM] [ML] Starting a job without enough memory doesn't always show Out of Memory error

[spong]

If a job fails to start due to a lack of memory this error should be presented to the user by means of an error toast. While this behavior is present in some cases (see #45316), there seems to be some inconsistency on whether or not an error is displayed.

In the below gif, you'll notice that the force_start_datafeed request returns an empty object instead of an error. In this instance, when we next refresh the state of all jobs, we can show that the job was not able to start due to a lack of memory by putting the error message/current state as hover text on the job (or potentially a callout at the top saying we're at max utilization).

State stuck as 'opening' in ML UI:

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[spong]

As introduced in #50766, looks like we can add even more details around the no node/OoM cases to better improve the UX here.

[spong]

@MadameSheema this is still relevant, and should be prioritized as we continue to add more ML jobs.

[cybersecdiva]

Tested in 8.9.0 BC5

Build Details:
VERSION: 8.9.0 BC5
BUILD: 64715
COMMIT: beb56356c5c037441f89264361302513ff5bd9f8

Preconditions:

• Kibana must be running
• ML node must be set up with the lowest memory (1 GB)

Describe the bug:
Starting an ML job without enough memory doesn't always show Out of Memory error

Steps to reproduce:

1. Navigate to Security—> Manage Rules—> ML job settings

2. Under ML job settings enable a minimum of 10 ML jobs

3. Save the change when the Pop-up notification displays to confirm changes

4. Reload the page for the change to take effect

5. Navigate to Security --> Manage --> Rules

Current behavior

No Out of Memory errors are warnings for ML jobs started (over 10+ jobs enabled)

Expected behavior:

No Out of Memory errors are warnings for ML jobs started (over 10+ jobs enabled)

Observations:

A review of the ML job memory usages shows that when there are more than 10+ jobs even 20 or more jobs on enabled on the cluster, there are no out of memory error limits displayed

Screenshots of behavior:

Machine Learning Memory Usage

Conclusion:

Behavior appears to be performing as expected. Validated ✅ bug is fiexed.

@MadameSheema and @spong FYI Updated observations