Threading issue -> crash TryCatch and isolate

[matiwinnetou]

We are observing the following crash on 4.8.0 release in production usually every 2-3 hours:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f13561f2a8d, pid=39, tid=0x00007f134cbf7700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libj2v8_linux_x86_64.so+0x5eba8d]  v8::TryCatch::TryCatch()+0x1d
#
# Core dump written. Default location: //core or core.39
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007f1230002000):  JavaThread "http-nio-8080-exec-9" daemon [_thread_in_native, id=146, stack(0x00007f134caf7000,0x00007f134cbf8000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Registers:
RAX=0x303035313a227461, RBX=0x00007f134cbf5700, RCX=0x00007f13e62a02c0, RDX=0x00007f134cbf7a30
RSP=0x00007f134cbf56a0, RBP=0x00007f134cbf56c0, RSI=0x0000000000000001, RDI=0x0000000000000020
R8 =0x0000000000000003, R9 =0x00001f703a104241, R10=0x00001f703a1aadb9, R11=0x00007f13e5a38f50
R12=0x0000000000000000, R13=0x00000000e50cc6c0, R14=0x00007f1268078ef0, R15=0x00007f1230002000
RIP=0x00007f13561f2a8d, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
  TRAPNO=0x000000000000000d

Top of Stack: (sp=0x00007f134cbf56a0)
0x00007f134cbf56a0:   00007f134cbf56c0 00007f13562146d1
0x00007f134cbf56b0:   00007f126807aab0 00007f126807aab0
0x00007f134cbf56c0:   00007f134cbf58d0 00007f135613aa04
0x00007f134cbf56d0:   00007f134cbf5a30 00007f134cbf58f0
0x00007f134cbf56e0:   00007f134cbf58f8 00007f134cbf5908
0x00007f134cbf56f0:   00007f12ec008390 00007f12300021f8
0x00007f134cbf5700:   303035313a227461 0000000000000000
0x00007f134cbf5710:   0000000400000003 0000000068060f00
0x00007f134cbf5720:   00007f1300000060 00007f12ec008390
0x00007f134cbf5730:   00007f1268060ff8 0000000000000000
0x00007f134cbf5740:   00007f1268060fe8 00007f1268061000
0x00007f134cbf5750:   00007f1268061038 ffffffff00000002
0x00007f134cbf5760:   00007f134cbf5790 00007f1268060fe8
0x00007f134cbf5770:   00007f1268061038 00007f1268060ff0
0x00007f134cbf5780:   00007f1268061038 00007f1230002000
0x00007f134cbf5790:   00007f1268060ff0 00007f135621471d
0x00007f134cbf57a0:   00007f1268060fe8 00007f135614d29f
0x00007f134cbf57b0:   00007f1268060fe8 00007f134cbf58b0
0x00007f134cbf57c0:   00007f134cbf5a20 00007f1356145f28
0x00007f134cbf57d0:   00007f1268060fe8 00007f12ec05c290
0x00007f134cbf57e0:   00007f1268060ff0 00007f12ec008330
0x00007f134cbf57f0:   00007f134cbf5a30 00007f12300021f8
0x00007f134cbf5800:   00007f1268060ff8 000000000000000b
0x00007f134cbf5810:   00007f1268061038 00007f1268061038
0x00007f134cbf5820:   00007f1268061038 00007f126807aab0
0x00007f134cbf5830:   00007f1268060ff8 00007f1268060ff8
0x00007f134cbf5840:   00007f12ec062770 00007f12ec062770
0x00007f134cbf5850:   00007f12ec062770 00007f12ec008390
0x00007f134cbf5860:   00007f1268078ef0 00007f12ec008390
0x00007f134cbf5870:   00007f1268060ff0 00007f12ec0626d0
0x00007f134cbf5880:   00007f12ec0626d0 00007f12ec0626d0
0x00007f134cbf5890:   00007f12ec008390 00007f12ec05bad0 

Instructions: (pc=0x00007f13561f2a8d)
0x00007f13561f2a6d:   f3 ff 90 55 48 89 e5 53 48 89 fb 48 83 ec 18 48
0x00007f13561f2a7d:   8b 05 a5 2c 03 01 8b 38 e8 e6 b5 f0 ff 48 89 03
0x00007f13561f2a8d:   48 8b 80 e0 8c 00 00 48 89 df 48 89 43 08 0f b6
0x00007f13561f2a9d:   43 28 83 e0 e0 83 c8 06 88 43 28 e8 f3 6c f3 ff 

Register to memory mapping:

RAX=0x303035313a227461 is an unknown value
RBX=0x00007f134cbf5700 is pointing into the stack for thread: 0x00007f1230002000
RCX=0x00007f13e62a02c0: <offset 0x2182c0> in /lib/x86_64-linux-gnu/libpthread.so.0 at 0x00007f13e6088000
RDX=0x00007f134cbf7a30 is pointing into the stack for thread: 0x00007f1230002000
RSP=0x00007f134cbf56a0 is pointing into the stack for thread: 0x00007f1230002000
RBP=0x00007f134cbf56c0 is pointing into the stack for thread: 0x00007f1230002000
RSI=0x0000000000000001 is an unknown value
RDI=0x0000000000000020 is an unknown value
R8 =0x0000000000000003 is an unknown value
R9 =0x00001f703a104241 is an unknown value
R10=0x00001f703a1aadb9 is an unknown value
R11=0x00007f13e5a38f50: <offset 0x194f50> in /lib/x86_64-linux-gnu/libc.so.6 at 0x00007f13e58a4000
R12=0x0000000000000000 is an unknown value
R13=0x00000000e50cc6c0 is an oop
[Ljava.lang.Object; 
 - klass: 'java/lang/Object'[]
 - length: 3
R14=0x00007f1268078ef0 is an unknown value
R15=0x00007f1230002000 is a thread


Stack: [0x00007f134caf7000,0x00007f134cbf8000],  sp=0x00007f134cbf56a0,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libj2v8_linux_x86_64.so+0x5eba8d]  v8::TryCatch::TryCatch()+0x1d
C  [libj2v8_linux_x86_64.so+0x533a04]  invokeFunction(JNIEnv_*, v8::Isolate*, long&, long&, _jstring*&, long&, v8::Local<v8::Value>&)+0x300
C  [libj2v8_linux_x86_64.so+0x533ed4]  Java_com_eclipsesource_v8_V8__1executeFunction__JIJLjava_lang_String_2J+0x192
J 10028  com.eclipsesource.v8.V8._executeFunction(JIJLjava/lang/String;J)Ljava/lang/Object; (0 bytes) @ 0x00007f13d2134e15 [0x00007f13d2134d40+0xd5]
J 11377 C2 com.eclipsesource.v8.V8Object.executeJSFunction(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/Object; (279 bytes) @ 0x00007f13d193f188 [0x00007f13d193ee40+0x348]
J 11579 C2 pubse.ecs.system.renderer.servlet.j2v8.PoolingJ2V8JavaScriptRenderer.apply(Lde/mobile/ecs/World;)V (340 bytes) @ 0x00007f13d242bc8c [0x00007f13d242b420+0x86c]
J 11594 C2 pubse.ecs.system.renderer.ResponseRenderer.apply(Lde/mobile/ecs/World;)V (79 bytes) @ 0x00007f13d2441740 [0x00007f13d24413a0+0x3a0]
J 11147 C2 com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed()Ljava/lang/Object; (91 bytes) @ 0x00007f13d23a8b64 [0x00007f13d23a2c60+0x5f04]
J 11512 C2 pubse.config.InboundModule$Monitor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object; (70 bytes) @ 0x00007f13d240ae1c [0x00007f13d240ad00+0x11c]
J 11147 C2 com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed()Ljava/lang/Object; (91 bytes) @ 0x00007f13d23a2de0 [0x00007f13d23a2c60+0x180]
J 11738 C2 pubse.tomcat.handler.ServletRouter$$Lambda$185.handle(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (10 bytes) @ 0x00007f13d254dd38 [0x00007f13d254d500+0x838]
J 12185 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f13d11a9104 [0x00007f13d11a8fa0+0x164]
J 11186 C2 org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (402 bytes) @ 0x00007f13d23c854c [0x00007f13d23c7200+0x134c]
J 12185 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f13d11a940c [0x00007f13d11a8fa0+0x46c]
J 11550 C2 pubse.util.ServletFilters$Utf8EncodingFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (21 bytes) @ 0x00007f13d2399980 [0x00007f13d23995a0+0x3e0]
J 12185 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f13d11a940c [0x00007f13d11a8fa0+0x46c]
J 11623 C2 pubse.util.ServletFilters$LoggingFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (76 bytes) @ 0x00007f13d245dd6c [0x00007f13d245da40+0x32c]
J 12185 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f13d11a940c [0x00007f13d11a8fa0+0x46c]
J 11543 C2 org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (1389 bytes) @ 0x00007f13d1dc6f2c [0x00007f13d1dc6d40+0x1ec]
J 11538 C2 org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (172 bytes) @ 0x00007f13d1e035ec [0x00007f13d1e03060+0x58c]
J 11851 C2 org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (903 bytes) @ 0x00007f13d207a6ec [0x00007f13d207a2c0+0x42c]
J 11733 C2 org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (402 bytes) @ 0x00007f13d25469b0 [0x00007f13d2546640+0x370]
J 11584 C2 org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (805 bytes) @ 0x00007f13d2445554 [0x00007f13d2445060+0x4f4]
J 11808 C2 org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1128 bytes) @ 0x00007f13d25793f8 [0x00007f13d2579180+0x278]
J 11805 C2 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1073 bytes) @ 0x00007f13d1f4a6dc [0x00007f13d1f4a200+0x4dc]
J 11444 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun()V (603 bytes) @ 0x00007f13d18b1250 [0x00007f13d18b10e0+0x170]
J 11438 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run()V (79 bytes) @ 0x00007f13d19f216c [0x00007f13d19f20c0+0xac]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x691d16]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x692221]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x6926c7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x72da50]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa76833]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7697c]  JavaThread::run()+0x11c
V  [libjvm.so+0x927568]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x76ba]  start_thread+0xca

[matiwinnetou]

While the original cause is not known, current code is also quite dangerous. Despite the fact that isolate object is available, the global one is used.

All the places can be changed in com_eclipsesource_v8_V8Impl.cpp to what is believed to be more thread safe code.

v8::TryCatch::TryCatch()
    : isolate_(i::Isolate::Current()),
      next_(isolate_->try_catch_handler()),
      is_verbose_(false),
      can_continue_(true),
      capture_message_(true),
      rethrow_(false),
      has_terminated_(false) {
  ResetInternal();
  // Special handling for simulators which have a separate JS stack.
  js_stack_comparable_address_ =
      reinterpret_cast<void*>(v8::internal::SimulatorStack::RegisterCTryCatch(
          isolate_, v8::internal::GetCurrentStackPosition()));
  isolate_->RegisterTryCatchHandler(this);
}


v8::TryCatch::TryCatch(v8::Isolate* isolate)
    : isolate_(reinterpret_cast<i::Isolate*>(isolate)),
      next_(isolate_->try_catch_handler()),
      is_verbose_(false),
      can_continue_(true),
      capture_message_(true),
      rethrow_(false),
      has_terminated_(false) {
  ResetInternal();
  // Special handling for simulators which have a separate JS stack.
  js_stack_comparable_address_ =
      reinterpret_cast<void*>(v8::internal::SimulatorStack::RegisterCTryCatch(
          isolate_, v8::internal::GetCurrentStackPosition()));
  isolate_->RegisterTryCatchHandler(this);
}

Rationale is that most likely the following code is returns null:

i::Isolate::Current()

It may seem therefore better to use a constructor to which isolate is explicitly passed rather than inferred via a static call.