Core dumped version 0.4.0 #193
Comments
|
I have totally the same issue! |
|
We need to find where this free is coming from, do you have a native backtrace at the point of failure? |
|
Hi @SamSaffron Used GBD to catch the native back trace: |
|
Oh ... is there any forking involved here? Did the parent process create an instance of |
|
Yes, we are doing forking because we use Puma as server. |
|
We are working on a fork safe method for mini racer, it just got upstreamed into libv8 a few months ago. In the mean time you must ensure that no contexts are created in your primary process. |
|
note, we are waiting on this commit to land in libv8-node v8/v8@a18a674#diff-a4b07237cb85c6098031ab6d84b4cdbd3fd31b919d4f89ab6e22efe7070a539f My guess is this will happen in the next release, the commit is from november. (fyi @lloeki ) |
|
Thanks @SamSaffron. Indeed the 16.x branch has node 9.0 https://github.com/nodejs/node/blob/v16.x/deps/v8/include/v8-version.h |
|
@SamSaffron. Thank you, Sam! |
|
Node 16 is out. I'm attempting a build of libv8-node 16.0.0. |
|
libv8-node built, artifact gems are downloadable on CI. The CI test phase against mini_racer fails though, TBH I did not investigate, but from a cursory look it involves v8 headers, so I think maybe code needs an update because of some breaking changes in the V8 API. |
|
Although I can now upgrade to mini_racer 0.5.0 (#218 (comment)) and libv8-node 16.10, I'm still having this issue too (since 0.4 so I am currently staying on 0.3 branch) and after reading this thread I'm not sure if there's anything I can do about it or help? or if there was a solution found somewhere? It occurs during my integrations tests, which are running using Capybara (server is Puma). |
You should check how you're running puma. If it's running in a single process, you should be fine actually. We use mini_racer heavily as part of our application logic. As long as you are not forking after using |
|
Mm ok, I don't intentionally thread at least but there's probably some libraries doing so, puma is started by Capybara I just set: I'm surprised the problem isn't more common so I suppose it's working correctly for most. What I just noticed now is that I have one machine where it's working fine, and another where it's not (exactly the same codebase, ruby version, etc..). The only difference I can think of so far is that the one working is running Ubuntu 21.10 and the one failing (free(): invalid pointer) is running Ubuntu Jan 4 edit: I tried updating all system libraries, re-installing all gems (and thus compiling native extensions) on my machine (even tried on a different version of ruby: 3.0.3 instead of 2.6.4), but the result is still the same 🕵️ I checked the differences in the compiled modules between my two machines, noticed just one small difference here (elk is the 20.04 machine where it's not working, desktop is the 21.10 machine where it's working fine). Both ruby are compiled The only notable difference I see here is "librt" which is absent in the second build but I doubt this has any link. Finally I tried re-installing ruby 3.0.3 without jemalloc to see if it changes anything, and yay 🙌 no exception! So naturally I tried re-installing with jemalloc to confirm if the issue was with jemalloc and OBVIOUSLY: still no exception... 1h later I try again and I get the exception again: (which now occurs every time, it's not random). After doing this, I noticed no problem with the ruby without jemalloc, I re-installed with jemalloc again, and the exception showed at first try. So I am not sure if that is connected but it certainly looks like it. (on my 21.10 machine I can't reproduce though, even with jemalloc). @thomasbalsloev @lephyrius where you using jemalloc in your Ruby install? |
|
This feels related to fork safety. Can you try again with 0.6.1 and specify |
|
@jarthod Yes, I believe so (PS: left my former position, so I can't check). |
|
@SamSaffron thanks for the response. So I just tried with 0.6.1 first to confirm I was still reproducing the issue (I am). And then I tried adding Also on my dev machine (running Ubuntu 21.10) on which I wasn't getting the crash before, it now also fails reliably on this line. I confirmed with some puts debugging: # rails_helper.rb
# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require 'spec_helper'
require_relative "../config/environment"
require 'rspec/rails'
# Add additional requires below this line. Rails is not loaded until this point!
puts '1 before'
MiniRacer::Platform.set_flags! :single_threaded
puts '2 after'
require 'capybara/rspec'
require 'capybara/cuprite'
require 'fabrication'
Capybara.javascript_driver = :cuprite
Capybara.server = :puma, { Silent: true }
Capybara.register_driver(:cuprite) do |app|
Capybara::Cuprite::Driver.new(app, js_errors: true, timeout: 10, process_timeout: 10)
end
# [...]Output: For the record I also tried with just Also I then tried to see if this was enough to reproduce the crash without ANYTHING else, and yes it is: > ruby -r mini_racer -e "MiniRacer::Platform.set_flags! :single_threaded"
free(): invalid pointer
Aborted (core dumped)So just requiring mini_racer (0.6.1) and then running this line causes my ruby (3.0.3 with jemalloc) to crash. > ruby -v
ruby 3.0.3p157 (2021-11-24 revision 3fb7d2cadc) [x86_64-linux]
> ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lpthread -lrt -lrt -ljemalloc -ldl -lcrypt -lmI also compared again this line on a ruby with and without jemalloc (two slightly different versions so it's easier to switch but the problem is the same on any version I tested) and it still shows that the problem only occurs with jemalloc: > rbenv shell 2.6.6
> ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lpthread -lrt -lrt -ldl -lcrypt -lm
> ruby -r mini_racer -e "puts MiniRacer::VERSION; MiniRacer::Platform.set_flags! :single_threaded"
0.6.1> rbenv shell 2.6.4
> ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lpthread -lrt -lrt -ljemalloc -ldl -lcrypt -lm
> ruby -r mini_racer -e "puts MiniRacer::VERSION; MiniRacer::Platform.set_flags! :single_threaded"
0.6.1
free(): invalid pointer
Aborted (core dumped) |
|
Quick check with Ruby 3.1.0 on Using Not sure if there is a significant difference between 0.6.1 and 0.6.2 but could you try just in case, @jarthod? |
|
Mm interesting, I don't often use LD_PRELOAD for jemalloc I tend to set it at compile time to make sure every processes uses it with: I just tried on my local 3.0.3 (which is compiled with jemalloc) with and without LD_PRELOAD (expecting nothing much but the same result as in both case it should use jemalloc), yet I don't: I then tried on a 3.1.0 freshly installed WITHOUT jemalloc compile option, and in this case it seems to be working ok with LD_PRELOAD: So it looks like using jemalloc dynamically is not causing trouble here, but compiling it with ruby is ( So at this point, it looks like ruby 3.1.0 might have changed something compared with all the previous version I got this error with (2.6.4, 2.6.6, 3.0.3). So I reseted everything (uninstalled ruby 3.0.3, 3.1.0), then re-installed each of these version Then I tried each: So at least I can reliably reproduce this behavior, I couldn't find anything in ruby changelog that would justify this, I'm not sure if it's just luck at this point. I'll try to upgrade one of my project to ruby 3.1 and mini_racer 0.6.2 to see if my CI is happy with it after running a dozen tests on it. |
|
That's very interesting, thank you for the analysis @jarthod! 🙇 I think the question is, if we want to tread jemalloc (either at compile time or via I see two options here (/cc @SamSaffron):
(we should probably do 1. in any case). |
|
Thanks @tisba, I have trouble understanding how jemalloc could cause this unless the allocator is improperly used by libv8 or mini_racer (relying implicitly or explicitly on some glibc specific quirks like over allocation). But in any case if you don't have any obvious idea of such weird usage I suppose we should start with option 1 of documenting this, It doesn't look like too many people are impacted by this considering the traffic on this thread (which is surprising to me). And maybe someone later will find more details or the actual reason. I don't have the courage to start diving more into GDB debugging and C internals at this point ^^. For my personal use-case, I'll stick with mini_racer 0.3 for projects which I can't upgrade to ruby 3.1 yet, and that's probably an acceptable workaround for most. |
|
I think the initial issue has been resolved, mini_racer can now be used in a fork-safe way, see https://github.com/rubyjs/mini_racer#fork-safety. I'll open a separate issue around @jarthod's jemalloc findings. |
Hi.
When having mini_racer version 0.4.0 in our gemset, our rspec tests fail hard with
No other clues are provided and I am not able to pinpoint the exact point of failure. The failure does not appear to originate from a specific test as it fails on a different one each time.
Ruby: ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
OS: Ubuntu 20.04.2 LTS (dockerized)
Rails: 6.0.3.6
When I regress to mini racer version 0.3.1 the failure does NOT happen.
I have tried to analyze the dumped core but don't have the experience to identify the problem from it.
Below is the ruby back trace gotten from using "gdbdump" on the dumped core. I don't know if that helps or not.
I'm prette much in the dark with this. Please let me know if you guys need more information.
Best regards, Thomas Balsløv.
The text was updated successfully, but these errors were encountered: