-
Notifications
You must be signed in to change notification settings - Fork 359
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Threading/Isolate issue ->Spontaneous JVM crash on creation of V8 #306
Comments
To some extend those issues do not surprise me, J2V8 has so far been primarily used for UI development and we operate a cluster of servers. Some errors occur only under heavy load with lots of users hitting (we have 1.500 req/s to the cluster). No bug can easily slip through this. |
Looks like Isolate::Current or Isolate::GetCurrent is deprecated and anti-pattern:
|
Most likely this exception is happening somewhere here: private internal isolate: isolate.cc
|
Reproduced on DEV BOX. The way we are able to reproduce is the following, we create and destroy J2V8 in a pool by 20 threads, previously allocating locks accordingly.
|
I got a core file, it has 26 GB though: |
|
|
@irbull Most likely entry_stack_ is NULL, on creation of V8 we should not even enter this if statement:
in the first place. Again this is the case where CurrentIsolate is not pointing to the right data, see #308 and #310. The correct way to fix all those problems I think would be to really make sure binary semaphores of J2V8 and Integer semaphores of V8 are properly synchronised. |
I will try to run the build with assertions on. |
12:50:00,833 [GeoLocationResolver RUNNING] INFO p.s.GeoLocationResolver Remote geo data loaded. With assertions on it is not even possible to debug...
|
I just returned from my summer vacation. I'm going through my backlog of emails, bugs, slack messages, etc.. I see you raised / commented on a bunch of issues @matiwinnetou. I will begin looking through these today. |
@irbull Thank you so much, I am willing to support you and help. I have been digging deep to this but my C++ knowledge is really rusty I must say. I provided lots of exceptions and stack traces and I hope insights into the problem |
I have been thinking that perhaps while in PROD this bug does not produce many issues for us, with easy workarounds of not recreating V8s very often I have to say that this bug maybe in fact hides a real problem that we are facing in #313. Anyway, for your info, I am able to reproduce this bug on a dev box (OSX, Sierra) on latest branch : "isolate_enter_branch" build quite quickly (as before). |
I will try to test this as well, tomorrow or on Monday if latest master build fixes this... |
I tested this now number of minutes (like 1 hour), I cannot reproduce this anymore. This seems fixed. I will reopen when I discover this again. |
Let me explain a bit setup.
We have a new page running, 100% J2V8 on the server with react and react on the client.
This renderer consists of J2V8 with apache commons pool and evictor running every 10 minutes. There is minimum 5 j2v8 that thread has to keep in pool. Generally it works but every now and then JVM keeps crashing.
and here info on stack trace, seems to be in native code
Make object is acquiring lock and releasing after v8 is created, all boxes have 5 GB ram for which 1-2 GB is reserved for JVM and the rest for native memory (V8). Since it was risky for us to move to J2V8 we decided to host a new use case on new set of boxes in production and observe what happens. On various machines we see sporadic crashes from time to time.
Commons pool evictor for the moment is ran every 10 mins and destroys and then creates J2V8 based on traffic on demand.
Make object is implemented as follows:
We observe those crashes on at least two JVM 1.8 versions:
1.8.0_40 and 1.8.0_131 64 bit on intel both running on linux ubuntu (xenial)
Any idea what could cause this? Is this some bug in our code or more likely in native C++ JNI bridge to V8?
Last but not least, the version of j2v8 we use is 4.8.0.
The text was updated successfully, but these errors were encountered: