HTTP2 client: IllegalStateException: Cannot release an already released entry

[lorban]

Jetty version
9.4.x; maybe 10.0.x and 11.0.x too

Description
There seems to be a race condition in the HTTP2 client that makes it very rarely fail with a IllegalStateException when multiplexing is enabled:

java.util.concurrent.CompletionException: java.lang.IllegalStateException: Cannot release an already released entry
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:783)
	at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
	at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
	at org.mortbay.jetty.load.generator.LoadGenerator.process(LoadGenerator.java:358)
	at org.mortbay.jetty.load.generator.LoadGenerator.lambda$begin$0(LoadGenerator.java:187)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot release an already released entry
	at org.eclipse.jetty.util.Pool$Entry.tryRelease(Pool.java:623)
	at org.eclipse.jetty.util.Pool.release(Pool.java:414)
	at org.eclipse.jetty.client.AbstractConnectionPool.deactivate(AbstractConnectionPool.java:378)
	at org.eclipse.jetty.client.AbstractConnectionPool.release(AbstractConnectionPool.java:364)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:352)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:336)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:315)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:309)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:286)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:263)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:600)
	at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:778)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:765)
	at org.mortbay.jetty.load.generator.LoadGenerator$Sender.send(LoadGenerator.java:641)
	at org.mortbay.jetty.load.generator.LoadGenerator$Sender.send(LoadGenerator.java:601)
	at org.mortbay.jetty.load.generator.LoadGenerator$Sender.access$500(LoadGenerator.java:562)
	at org.mortbay.jetty.load.generator.LoadGenerator.sendResourceTree(LoadGenerator.java:449)
	at org.mortbay.jetty.load.generator.LoadGenerator.process(LoadGenerator.java:334)
	... 5 more

This was first seen in jetty-load-generator's HTTP2WebsiteLoadGeneratorTest.testHTTP2WithPush() test but it seems generic, and just hard to trigger.

[lorban]

The problem seems to be linked to the natural race that exists between a normal request/response cycle terminating and that cycle timing out: both are going to race to release the multiplexed connection back into the pool.

The root cause seems to be in HttpDestination.release(Connection) as it contains a pair of non-atomic tests:

if (connectionPool.isActive(connection))
    if (connectionPool.release(connection))

The connection might have been released in between these two tests by the other thread which would make both the request processing thread as well as the timeout thread try to release the connection to the pool which gets sanctioned by a IllegalStateException.