Unusual high number of 'sun.nio.ch.EPollArrayWrapper.epollWait' threads in runnable state

[ajames-branch]

One of our spark streaming application is seeing unusual high number of following threads in RUNNABLE state. I have seen as high as 22K such threads for a process. The number keeps on increasing and never comes down. Eventually the process is oom-killed by kernel. Java native memory tracker showed that memory allocated to threads were ever increasing. I suspect the following thread is causing memory leak eventually leading to oom killer.

`priority:5 - threadId:0x00007fde00075800 - nativeId:0x1df6 - nativeId (decimal):7670 - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x000000079c648758> (a sun.nio.ch.Util$3)
• locked <0x000000079c648748> (a java.util.Collections$UnmodifiableSet)
• locked <0x000000079c648630> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
  at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
  at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
  at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
  at java.lang.Thread.run(Thread.java:748)`

[sbordet]

What Jetty version?

Jetty creates a fixed number of selectors, where the number is either explicitly configured by users, or defaulted with some heuristic based on the number of cores.
22K selectors means that something is continuously creating selectors and starting them, without stopping them (which would get rid of them).

A wild guess is that you create multiple instances of HttpClient (which in turn create selectors), you start them, but you don't stop them.

[ajames-branch]

Thank You @sbordet for the pointers. Let me check and get back.

[sbordet]

@ajames-branch news?

[YeTingGe]

I have similar issue with vertx+netty
In one of my env, the thread number keep increasing when I use vertx 3.8.3
When I swtich back to 3.7.1, everthing seems fine.

"vert.x-eventloop-thread-1" #21626 prio=5 os_prio=0 tid=0x00007fa8997cd000 nid=0x3fbe runnable [0x00007fa77e8b6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005ddb48e28> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005ddb38bc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005ddb38aa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:824)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

"vert.x-eventloop-thread-0" #21625 prio=5 os_prio=0 tid=0x00007fa8997ca800 nid=0x3fbd runnable [0x00007fa77e9b7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000005ddb352e0> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000005ddb25078> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000005ddb
"vertx-blocked-thread-checker" #21623 daemon prio=5 os_prio=0 tid=0x00007fa89977d800 nid=0x3fbc in Object.wait() [0x00007fa77eab8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000005ddb22370> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

[sbordet]

Closing as no further information was provided.
The vert.x comment is not relevant to Jetty.