Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace PromiseCombiner usage on ClientConnectionsShutdown #1487

Merged
merged 3 commits into from
Mar 2, 2023

Conversation

jguerra
Copy link
Collaborator

@jguerra jguerra commented Mar 1, 2023

ClientConnectionsShutdown is broken in two different ways

  1. When gracefullyShutdownClientChannels() is called as part of a discovery callback, the use of ImmediateEventExecutor.INSTANCE triggers a deadlock check in netty:
Caused by: io.netty.util.concurrent.BlockingOperationException: DefaultPromise@7ff6ea19(incomplete)
	at app//io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:463)
	at app//io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:687)
	at app//io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:295)
  1. The use of PromiseCombiner isn't thread safe and the future isn't always finished correctly

I replaced the use of PromiseCombiner with bulk operations on the ChannelGroup, and also added unit tests


//racy situation if new connections are still coming in, but any channels created after newCloseFuture will
//be closed during the force close stage
ChannelGroupFuture closeFuture = channels.newCloseFuture();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much cleaner!

}
});
// Wait for all the attempts to close connections gracefully, or max of 30 secs
if(closeFuture.await(30, TimeUnit.SECONDS)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, this would end up waiting a max of 30s for shutdown across all channels. While it's an improvement, I think it's a change from the existing behavior which waits significantly longer.
Can we make this configurable as we roll this out?

…onto jvm shutdown hook thread. Make force close timeout configurable
closeFuture.addListener(future -> {
if (!timeoutTask.isDone()) {
//close happened before the timeout
LOG.info("All connections closed within timeout");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: do we care? 😉

@jguerra jguerra merged commit 2019474 into master Mar 2, 2023
@jguerra jguerra deleted the jg/client_shutdown branch March 2, 2023 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants