Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stopping informer does not cancel the resyncFuture nor it shutdown the resyncExecutor #3014

Closed
akram opened this issue Apr 20, 2021 · 1 comment · Fixed by #3017
Closed
Assignees

Comments

@akram
Copy link
Contributor

akram commented Apr 20, 2021

While trying to stop a set of informers and start new ones, there is remaining task trying to schedule in an executor:

2021-04-19 16:14:50 SEVERE  hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException A thread (informer-controller-ImageStream/460) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.

Incidently with the error, I have several errors of this type showing that a task is still trying to be submitted on an open executor.

2021-04-19 16:15:01 WARNING io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener onFailure Exec Failure
java.io.InterruptedIOException: executor rejected
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:186)
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:137)
	at okhttp3.RealCall.enqueue(RealCall.java:127)
	at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:193)
[...]
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@6285bec1 rejected from java.util.concurrent.ThreadPoolExecutor@509c2579[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:183)
	... 104 more

After investigations, I see that the resyncFuture https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/cache/Controller.java#L103 of informers are not canceled not their executors are shutdown.

@akram
Copy link
Contributor Author

akram commented Apr 20, 2021

fixed by #3017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant