Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

Segfault 11 on 0.10.31 #8208

Closed
lloydwatkin opened this issue Aug 20, 2014 · 49 comments
Closed

Segfault 11 on 0.10.31 #8208

lloydwatkin opened this issue Aug 20, 2014 · 49 comments

Comments

@lloydwatkin
Copy link

We're seeing this issue on 0.10.31 (but not 0.10.30) on both travis and OSX. I've used segfault-handler to get a trace. Hopefully this is the correct information you need and the correct place to raise this issue (i.e. its related to node and not a module).

PID 5799 received SIGSEGV for address: 0x8
0   segfault_handler.node               0x0000000100ce949f _ZL16segfault_handleriP9__siginfoPv + 287
1   libsystem_platform.dylib            0x00007fff87e8d5aa _sigtramp + 26
2   node                                0x000000010037acd4 _ZN2v88internal9Assembler15RecordRelocInfoENS0_9RelocInfo4ModeEl + 68
3   node                                0x00000001002293c0 _ZN2v88internal17BoundsCheckBbData10CoverCheckEPNS0_12HBoundsCheckEi + 192
4   node                                0x00000001001f86e4 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 308
5   node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
6   node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
7   node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
8   node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
9   node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
10  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
11  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
12  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
13  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
14  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
15  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
16  node                                0x00000001001f8854 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEPNS0_11HBasicBlockEPNS0_16BoundsCheckTableE + 676
17  node                                0x00000001001f7e55 _ZN2v88internal6HGraph30EliminateRedundantBoundsChecksEv + 229
18  node                                0x00000001001f7d48 _ZN2v88internal6HGraph8OptimizeEPNS0_17SmartArrayPointerIcEE + 1192
19  node                                0x000000010015d24a _ZN2v88internal18OptimizingCompiler13OptimizeGraphEv + 42
20  node                                0x000000010015fa16 _ZN2v88internalL12GenerateCodeEPNS0_15CompilationInfoE + 134
21  node                                0x000000010015e694 _ZN2v88internal8Compiler11CompileLazyEPNS0_15CompilationInfoE + 404
22  node                                0x00000001002a5322 _ZN2v88internal10JSFunction16CompileOptimizedENS0_6HandleIS1_EENS0_9BailoutIdENS0_18ClearExceptionFlagE + 162
23  node                                0x00000001003188e4 _ZN2v88internal36Runtime_CompileForOnStackReplacementENS0_9ArgumentsEPNS0_7IsolateE + 644
24  ???                                 0x0000296cbee06362 0x0 + 45547035583330
25  ???                                 0x0000296cbee132ad 0x0 + 45547035636397
26  ???                                 0x0000296cbf85db1c 0x0 + 45547046427420
@indutny
Copy link
Member

indutny commented Aug 20, 2014

Bad news, this is that CVE patch. cc @tjfontaine

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Hm... it is really odd that _ZN2v88internal9Assembler15RecordRelocInfoENS0_9RelocInfo4ModeEl is actually invoked at this place.

Is there any way to reproduce this segfault?

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Or can I connect to that machine and take a look at the core file?

@lloydwatkin
Copy link
Author

Unfortunately its private code so I can't share. I've got another system which uses some of the same libraries but the issue isn't occurring there.

Some of the packages used in our application are: xmpp-ftw, massah, express. xmpp-ftw makes use of a few libraries which build native extensions.

If there's any additional information I can provide please let me know and I will do what I can.

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Hm... Could you please gist a disassembly of _ZN2v88internal17BoundsCheckBbData10CoverCheckEPNS0_12HBoundsCheckEi. You could just run: lldb -- node and then disas -n _ZN2v88internal17BoundsCheckBbData10CoverCheckEPNS0_12HBoundsCheckEi.

@lloydwatkin
Copy link
Author

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Hm... May I ask you to build node.js in debug mode and try reproducing this issue again?

BUILDTYPE=Debug make -C out/ -j24 in a node.js folder.

@indutny
Copy link
Member

indutny commented Aug 20, 2014

And the binary will be in ./out/Debug/node.

@lloydwatkin
Copy link
Author

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Thank you, this is much more informative :)

@indutny
Copy link
Member

indutny commented Aug 20, 2014

One last thing, could you please gist a _ZNK2v88internal11HBasicBlock12IsStartBlockEv disassembly?

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Hm... it seems that that segv module isn't properly unwinding stack. Could you please turn on core dumps with ulimit -c unlimited and run your tests again? It should produce some /core/... files, which could be opened with lldb -c /core/...

@lloydwatkin
Copy link
Author

@lloydwatkin
Copy link
Author

Sorry I don't understand how to get the core files for you? Or how to generate them. I'm not familiar with lldb at all.

@indutny
Copy link
Member

indutny commented Aug 20, 2014

  1. ulimit -c unlimited
  2. ./out/Debug/node npm test
  3. ls /cores/
  4. lldb -c /cores/core.some-number
  5. bt

@cressie176
Copy link

Steps to reproduce...

git clone https://github.com/acuminous/yadda.git
cd yadda
npm install
npm test

The tests run until my Hospital scenario, then boom!

  FileSearch
    ✓ should return all files by default 
    ✓ should return the matching files when a regex is specified 
    ✓ should ignore missing search paths 

  Hospital
make: *** [test] Segmentation fault: 11
npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

I (and travis) get segmentation fault 11 consistently. Downgrading to 0.10.30 and the tests pass

You can see the segmentation fault here

@indutny
Copy link
Member

indutny commented Aug 20, 2014

Thanks!

Here is a better stack trace:

* thread #1: tid = 0x0000, 0x0000000100341efc node`v8::internal::HBasicBlock::block_id(this=0x0000000000000000) const + 12 at hydrogen.h:61, stop reason = signal SIGSTOP
  * frame #0: 0x0000000100341efc node`v8::internal::HBasicBlock::block_id(this=0x0000000000000000) const + 12 at hydrogen.h:61
    frame #1: 0x00000001003421a5 node`v8::internal::HBasicBlock::IsStartBlock(this=0x0000000000000000) const + 21 at hydrogen.h:98
    frame #2: 0x0000000100338ae0 node`v8::internal::HInstruction::InsertAfter(this=0x000000010183e2a0, previous=0x000000010183d150) + 272 at hydrogen-instructions.cc:598
    frame #3: 0x00000001003863ec node`v8::internal::BoundsCheckBbData::CoverCheck(this=0x0000000101849a40, new_check=0x000000010183e2a0, new_offset=0) + 540 at hydrogen.cc:3514
    frame #4: 0x000000010036080d node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x000000010182a9f0, table=0x00007fff5fbfc920) + 525 at hydrogen.cc:3654
    frame #5: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x000000010182b5a0, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #6: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101829a98, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #7: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101829fb8, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #8: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x00000001018292b0, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #9: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101828000, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #10: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x00000001018289b8, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #11: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x00000001018277f0, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #12: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101826690, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #13: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x00000001018251d0, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #14: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101825690, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #15: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x0000000101822a20, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #16: 0x0000000100360a2f node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478, bb=0x00000001018226b0, table=0x00007fff5fbfc920) + 1071 at hydrogen.cc:3675
    frame #17: 0x000000010035fed9 node`v8::internal::HGraph::EliminateRedundantBoundsChecks(this=0x0000000101822478) + 89 at hydrogen.cc:3693
    frame #18: 0x000000010035fe27 node`v8::internal::HGraph::Optimize(this=0x0000000101822478, bailout_reason=0x00007fff5fbfcb20) + 999 at hydrogen.cc:3359
    frame #19: 0x000000010026aac9 node`v8::internal::OptimizingCompiler::OptimizeGraph(this=0x00007fff5fbfcb78) + 217 at compiler.cc:359
    frame #20: 0x000000010026dc62 node`v8::internal::MakeCrankshaftCode(info=0x00007fff5fbfcd28) + 82 at compiler.cc:209
    frame #21: 0x000000010026d7f0 node`v8::internal::GenerateCode(info=0x00007fff5fbfcd28) + 96 at compiler.cc:393
    frame #22: 0x000000010026ae66 node`v8::internal::MakeCode(info=0x00007fff5fbfcd28) + 134 at compiler.cc:409
    frame #23: 0x000000010026c31c node`v8::internal::Compiler::CompileLazy(info=0x00007fff5fbfcd28) + 332 at compiler.cc:815
    frame #24: 0x00000001004442a3 node`v8::internal::CompileLazyHelper(info=0x00007fff5fbfcd28, flag=CLEAR_EXCEPTION) + 195 at objects.cc:7563
    frame #25: 0x0000000100444d66 node`v8::internal::JSFunction::CompileOptimized(function=Handle<v8::internal::JSFunction> at 0x00007fff5fbfce08, osr_ast_id=BailoutId at 0x00007fff5fbfce00, flag=CLEAR_EXCEPTION) + 102 at objects.cc:7666
    frame #26: 0x00000001004e7da5 node`v8::internal::Runtime_CompileForOnStackReplacement(args=Arguments at 0x00007fff5fbfd2d0, isolate=0x0000000101807800) + 1397 at runtime.cc:8185
(lldb)

@cressie176
Copy link

WRT yadda, the problem manifests itself here between lines 63 and 70. Which is a nested for loop used to calculate the Levenshtein distance in order to compare the similarity of two strings

@lloydwatkin
Copy link
Author

Yeah the original issue came from an application which used yadda at some level so I'm glad there's a public test case. I'm travelling the next two days so if there's no info forthcoming in the meantime I can generate data then.

@indutny
Copy link
Member

indutny commented Aug 21, 2014

I think I am getting some clearer understanding of it now, will try to figure it out ASAP.

@indutny
Copy link
Member

indutny commented Aug 21, 2014

This is a half-way fix:

diff --git a/deps/v8/src/hydrogen.cc b/deps/v8/src/hydrogen.cc
index 50d8e49..2621b30 100644
--- a/deps/v8/src/hydrogen.cc
+++ b/deps/v8/src/hydrogen.cc
@@ -3531,7 +3531,8 @@ class BoundsCheckBbData: public ZoneObject {
     lower_check_(lower_check),
     upper_check_(upper_check),
     next_in_bb_(next_in_bb),
-    father_in_dt_(father_in_dt) { }
+    father_in_dt_(father_in_dt) {
+  }

  private:
   BoundsCheckKey* key_;
@@ -3581,7 +3582,6 @@ class BoundsCheckBbData: public ZoneObject {
                     HBoundsCheck* tighter_check) {
     ASSERT(original_check->length() == tighter_check->length());
     MoveIndexIfNecessary(tighter_check->index(), original_check, tighter_check);
-    original_check->ReplaceAllUsesWith(original_check->index());
     original_check->SetOperandAt(0, tighter_check->index());
   }
 };
@@ -3624,7 +3624,9 @@ void HGraph::EliminateRedundantBoundsChecks(HBasicBlock* bb,
                                             BoundsCheckTable* table) {
   BoundsCheckBbData* bb_data_list = NULL;

-  for (HInstruction* i = bb->first(); i != NULL; i = i->next()) {
+  HInstruction* next;
+  for (HInstruction* i = bb->first(); i != NULL; i = next) {
+    next = i->next();
     if (!i->IsBoundsCheck()) continue;

     HBoundsCheck* check = HBoundsCheck::cast(i);

@indutny
Copy link
Member

indutny commented Aug 21, 2014

And here is the fix that works:

diff --git a/deps/v8/src/hydrogen.cc b/deps/v8/src/hydrogen.cc
index 50d8e49..7c8da73 100644
--- a/deps/v8/src/hydrogen.cc
+++ b/deps/v8/src/hydrogen.cc
@@ -3546,7 +3546,11 @@ class BoundsCheckBbData: public ZoneObject {
   void MoveIndexIfNecessary(HValue* index_raw,
                             HBoundsCheck* insert_before,
                             HInstruction* end_of_scan_range) {
-    ASSERT(index_raw->IsAdd() || index_raw->IsSub());
+    if (!index_raw->IsAdd() && !index_raw->IsSub()) {
+       // index_raw can be HAdd(index_base, offset), HSub(index_base, offset),
+       // or index_base directly. In the latter case, no need to move anything.
+       return;
+    }
     HBinaryOperation* index =
         HArithmeticBinaryOperation::cast(index_raw);
     HValue* left_input = index->left();
@@ -3581,7 +3585,6 @@ class BoundsCheckBbData: public ZoneObject {
                     HBoundsCheck* tighter_check) {
     ASSERT(original_check->length() == tighter_check->length());
     MoveIndexIfNecessary(tighter_check->index(), original_check, tighter_check);
-    original_check->ReplaceAllUsesWith(original_check->index());
     original_check->SetOperandAt(0, tighter_check->index());
   }
 };
@@ -3624,7 +3627,9 @@ void HGraph::EliminateRedundantBoundsChecks(HBasicBlock* bb,
                                             BoundsCheckTable* table) {
   BoundsCheckBbData* bb_data_list = NULL;

-  for (HInstruction* i = bb->first(); i != NULL; i = i->next()) {
+  HInstruction* next;
+  for (HInstruction* i = bb->first(); i != NULL; i = next) {
+    next = i->next();
     if (!i->IsBoundsCheck()) continue;

     HBoundsCheck* check = HBoundsCheck::cast(i);

Could you please give it a try?

@aredridel
Copy link

I'm giving this a try -- it crashes for me too.

@aredridel
Copy link

This fixes it for me.

@indutny
Copy link
Member

indutny commented Aug 21, 2014

cc @tjfontaine

@dshaw
Copy link

dshaw commented Sep 10, 2014

@indutny Did you mean to close this issue?

@indutny
Copy link
Member

indutny commented Sep 10, 2014

Yeah, when I have committed: 3122e0e

@sdneon
Copy link

sdneon commented Sep 11, 2014

@indutny thanks for the quick reply. Have found the discrepancy. Upon retesting with Release mode node instead with the patch, it works. My previous tests of the patch were with Debug mode node, and that is still crashing.

@mvdwalle
Copy link

So when will this get released? Projects are breaking all over ...

@tjfontaine
Copy link

we are in the release process at the moment

@adamkdean
Copy link

Any ETA on 0.10.32?

Nevermind, http://nodejs.org/dist/v0.10.32/

@aaparmar
Copy link

0.10.32 fixes the segfault problem 👍

@lukewilliamboswell
Copy link

Life saver! Thank you

@indutny
Copy link
Member

indutny commented Nov 15, 2014

@tjfontaine what do you think about unpublishing this version?

mscdex pushed a commit to mscdex/node that referenced this issue Dec 25, 2014
fd80a31 has introduced a segfault
during redundant boundary check elimination (nodejs#8208).

The problem consists of two parts:

  1. Abscense of instruction iterator in
     `EliminateRedundantBoundsChecks`. It was present in recent v8, but
     wasn't considered important at the time of backport. However, since
     the function is changing instructions order in block, it is
     important to not rely at `i->next()` at the end of the loop.
  2. Too strict ASSERT in `MoveIndexIfNecessary`. It is essentially a
     backport of a45c96ab from v8's upstream. See
     v8/v8@a45c96ab for details.

fix nodejs#8208
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests