[CI] Failure in {p0=ml/data_frame_analytics_crud/Test put regression given negative lambda}

[dakrone]

Looks like the cleanup from clearing ILM policies:

java.lang.RuntimeException: failed to delete policy: ml-size-based-ilm-policy
	at __randomizedtesting.SeedInfo.seed([5903DB6DE4744D8C:D157E4B74A882074]:0)
	at org.elasticsearch.test.rest.ESRestTestCase.lambda$deleteAllILMPolicies$7(ESRestTestCase.java:759)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.elasticsearch.test.rest.ESRestTestCase.deleteAllILMPolicies(ESRestTestCase.java:755)
	at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:576)
	at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:295)
...
Caused by: org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:34560], URI [/_ilm/policy/ml-size-based-ilm-policy], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Cannot delete policy [ml-size-based-ilm-policy]. It is in use by one or more indices: [.ml-stats-write]"}],"type":"illegal_argument_exception","reason":"Cannot delete policy [ml-size-based-ilm-policy]. It is in use by one or more indices: [.ml-stats-write]"},"status":400}
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
	at org.elasticsearch.test.rest.ESRestTestCase.lambda$deleteAllILMPolicies$7(ESRestTestCase.java:757)
	... 45 more

Failure is at: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request-2/20842/testReport/junit/org.elasticsearch.xpack.test.rest/XPackRestIT/test__p0_ml_data_frame_analytics_crud_Test_put_regression_given_negative_lambda_/

I believe the ML-installed ILM policy needs to be added to this list:

elasticsearch/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java

Lines 463 to 465 in 4eaf2cb

	protected Set<String> preserveILMPolicyIds() {
	return Sets.newHashSet("ilm-history-ilm-policy", "slm-history-ilm-policy", "watch-history-ilm-policy");
	}

So that it doesn't need to be cleared between tests

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[costin]

This issue is showing up in various places, e.g.
https://gradle-enterprise.elastic.co/s/slg3bb53z3jrg/tests/fcijb24gqtmna-vijn253lceb7m?open-stacktraces=WzFd

:x-pack:plugin:integTestRunner » org.elasticsearch.xpack.test.rest.XPackRestIT » test {p0=ml/inference_processor/Test create and delete pipeline with inference processor} (0.752s)
failed to delete policy: ml-size-based-ilm-policy
java.lang.RuntimeException: failed to delete policy: ml-size-based-ilm-policyOpen stacktrace
Caused by: org.elasticsearch.client.ResponseException: 
method [DELETE], host [http://127.0.0.1:36487], URI [/_ilm/policy/ml-size-based-ilm-policy], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Cannot delete policy [ml-size-based-ilm-policy]. It is in use by one or more indices: [.ml-stats-000001]"}],"type":"illegal_argument_exception","reason":"Cannot delete policy [ml-size-based-ilm-policy]. It is in use by one or more indices: [.ml-stats-000001]"},"status":400}
Close stacktrace

I also got bit by it locally through:

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=transform/transforms_start_stop/Test stop transform with force and wait_for_checkpoint true} FAILED
    java.lang.RuntimeException: failed to delete policy: ml-size-based-ilm-policy
        at __randomizedtesting.SeedInfo.seed([93A3953EA9ECD860:1BF7AAE40710B598]:0)