Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[0.9.0] Too fast leadership changes in overlord causes multiple leaders and service failure #3046

Closed
drcrallen opened this issue Jun 1, 2016 · 3 comments · Fixed by #3050
Closed

[0.9.0] Too fast leadership changes in overlord causes multiple leaders and service failure #3046

drcrallen opened this issue Jun 1, 2016 · 3 comments · Fixed by #3050
Labels
Bug

Comments

@drcrallen
Copy link
Contributor

It is possible for two overlords in HA mode to get into some sort of leadership fight which causes high cpu load on both instances and general instability of the indexing service.

Logs from overlord 1:

2016-06-01T17:55:33,794 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy - Started Resource Management Scheduler
2016-06-01T17:55:33,794 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Bowing out!
2016-06-01T17:55:33,798 ERROR [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Failed to lead: {class=io.druid.indexing.overlord.TaskMaster, exceptionType=class java.lang.reflect.I
nvocationTargetException, exceptionMessage=null}
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4614612 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2f19712b[Shutting down, pool size = 1, active threads = 0, queued tasks = 2, completed tasks = 966]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 19 more
2016-06-01T17:55:33,809 ERROR [Curator-LeaderSelector-0] org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:150) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4614612 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2f19712b[Shutting down, pool size = 1, active threads = 0, queued tasks = 2, completed tasks = 966]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
2016-06-01T17:55:44,008 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!

Logs from overlord 2:

2016-06-01T17:55:25,231 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy - Started Resource Management Scheduler
2016-06-01T17:55:25,231 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Bowing out!
2016-06-01T17:55:25,231 ERROR [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Failed to lead: {class=io.druid.indexing.overlord.TaskMaster, exceptionType=class java.lang.reflect.InvocationTargetException, exceptionMessage=null}
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1ade306d rejected from java.util.concurrent.ScheduledThreadPoolExecutor@20d11e35[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17550]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 19 more
2016-06-01T17:55:25,232 ERROR [Curator-LeaderSelector-0] org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:150) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1ade306d rejected from java.util.concurrent.ScheduledThreadPoolExecutor@20d11e35[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17550]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
2016-06-01T17:55:33,810 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!
@drcrallen drcrallen added the Bug label Jun 1, 2016
@gianm
Copy link
Contributor

gianm commented Jun 1, 2016

@drcrallen does the [0.9.0] tag mean this is a regression in 0.9.0 or just that this is the version you noticed it in?

@drcrallen
Copy link
Contributor Author

It is the version I noticed it in.

@drcrallen
Copy link
Contributor Author

Didn't want people to think it was a 0.9.1 thing :)

@drcrallen drcrallen mentioned this issue Jun 1, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make TaskMaster giant lock fair metamx/druid
2 participants
@gianm @drcrallen