Open connections increasing without cleaning up

[weberr13]

I am using consul to maintain a resource lock. My basic logic is:

run, Consul.newClient(), and cache it
run consul.sessionClient.createSession() and cache it
run consul.keyValueClient().aquireLock()

once that is done I have a loop that every second runs:
consul.agentClient().ping()
consul.sessionClient().renewSession(...)
consul.keyValueClient().getSession(key)
... .isPresent()
.. .get() == session from above

This all works great in go, but in java I see the connections to consul's port go up, down, then up again, then up without bound (up to 1000's of connections) before consul stops accepting connections. It is almost as if the WebTargets from getValue() calls are being persisted by the java garbage collector and since WebTarget does not implement a "close" method eventually java persists hundreds of connections. I am working on trying out this theory but I was hoping that you already had an approach to solve this.

[rickfast]

We have had that problem actually. Calling .ping() (which in turn hits the /agent/self) endpoint even at a slow-ish pace can eventually cause Consul to stop accepting connections. We think the problem is on the Consul side and that Consul is doing something weird on this endpoint.

Can you try your logic without agentClient().ping() and let me know if that fixes it?

[rickfast]

I just read this:

Response is auto-closed if we have Response.readEntity where the class
type is not InputStream.

BUT - ping() just calls .getStatusInfo() on the CXF response. I'm going to push a change to ping... would you mind building locally and trying it out?

[rickfast]

here's the fix: 1468aa8

[tartale]

@rickfast I'm one of @weberr13 's colleagues, trying to drop the changes in right now... stay tuned!

[tartale]

@rickfast The ping change looked like it worked; I applied the same logic to the session renewal (which had the same issue) and put up a PR.

With both changes, this issue would be resolved for us

[rickfast]

Cool. Nice find. I'll go in and find all of the other possible places the connections aren't being closed. I will cut you guys a release (0.9.7) right now.

[rickfast]

0.9.7 is now available in JCenter and Maven Central

[tartale]

cool thanks dude!