PyTorchModelIT testUpdateDeployment_GivenAllocationsAreIncreased failure

[davidkyle]

In #120064 a test failed due to an unusual cluster state update failure

[2025-01-29T11:27:29,499][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [javaRestTest-2] fatal error in thread [elasticsearch[javaRestTest-2][masterService#updateTask][T#1]], exiting
java.lang.AssertionError: java.lang.IllegalStateException: cluster state update executor did not preserve version numbers: [update model deployment]
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1082) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1032) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:244) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1685) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1682) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1277) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:452) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1256) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]
Caused by: java.lang.IllegalStateException: cluster state update executor did not preserve version numbers: [update model deployment]
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1080) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	... 13 more

The failure is fairly reproducible if the entire test suite is run not just a single test

./gradlew :x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest --tests "org.elasticsearch.xpack.ml.integration.PyTorchModelIT"

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)