Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LLVM MemorySSA assertion failure when running with ASAN: "Not the same accesses in the same order" #50170

Closed
topolarity opened this issue Jun 14, 2023 · 3 comments
Closed

LLVM MemorySSA assertion failure when running with ASAN: "Not the same accesses in the same order" #50170

topolarity opened this issue Jun 14, 2023 · 3 comments

Comments

@topolarity
Copy link
Member

topolarity commented Jun 14, 2023 •
edited
Loading

On my system (Ubuntu 22.04 WSL2) the following changes are needed for ASAN support to build locally (due partially to #47698):

diff --git a/src/dlload.c b/src/dlload.c
index 4e9e9c9ce4..a01f4f5ea6 100644
--- a/src/dlload.c
+++ b/src/dlload.c
@@ -188,7 +188,7 @@ JL_DLLEXPORT JL_NO_SANITIZE void *jl_dlopen(const char *filename, unsigned flags
         dlopen = (dlopen_prototype*)dlsym(RTLD_NEXT, "dlopen");
         if (!dlopen)
             return NULL;
-        void *libdl_handle = dlopen("libdl.so", RTLD_NOW | RTLD_NOLOAD);
+        void *libdl_handle = dlopen("libdl.so.2", RTLD_NOW | RTLD_NOLOAD);
         assert(libdl_handle);
         dlopen = (dlopen_prototype*)dlsym(libdl_handle, "dlopen");
         dlclose(libdl_handle);
diff --git a/src/julia.h b/src/julia.h
index 2140b0ad0a..f9ed65a729 100644
--- a/src/julia.h
+++ b/src/julia.h
@@ -2118,13 +2118,13 @@ void (ijl_longjmp)(jmp_buf _Buf, int _Value);
 #define jl_setjmp_name "sigsetjmp"
 #endif
 #define jl_setjmp(a,b) sigsetjmp(a,b)
-#if defined(_COMPILER_ASAN_ENABLED_) && __GLIBC__
-// Bypass the ASAN longjmp wrapper - we're unpoisoning the stack ourselves.
-JL_DLLIMPORT int __attribute__ ((nothrow)) (__libc_siglongjmp)(jl_jmp_buf buf, int val);
-#define jl_longjmp(a,b) __libc_siglongjmp(a,b)
-#else
+//#if defined(_COMPILER_ASAN_ENABLED_) && __GLIBC__
+//// Bypass the ASAN longjmp wrapper - we're unpoisoning the stack ourselves.
+//JL_DLLIMPORT int __attribute__ ((nothrow)) (__libc_siglongjmp)(jl_jmp_buf buf, int val);
+//#define jl_longjmp(a,b) __libc_siglongjmp(a,b)
+//#else
 #define jl_longjmp(a,b) siglongjmp(a,b)
-#endif
+//#endif
 #endif

If I then build with contrib/asan/build.sh ./tmp/test-asan -j4 debug the terminal is filled with JIT session error: Duplicate section.

The build completes, but the REPL crashes if you start Julia and hit Enter:

$ ./tmp/test-asan/asan/julia
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
              _
  _       _ _(_)_     |  Documentation: https://docs.julialang.org
 (_)     | (_) (_)    |
  _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
 | | | | | | |/ _` |  |
 | | |_| | | | (_| |  |  Version 1.10.0-DEV.1488 (2023-06-14)
_/ |\__'_|_|_|\__'_|  |  Commit 0fb86dd5c5* (0 days old master)
|__/                   |

JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia> JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia>
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia: /workspace/srcdir/llvm-project/llvm/lib/Analysis/MemorySSA.cpp:2079: void llvm::MemorySSA::verifyOrderingDominationAndDefUses(llvm::Function&, llvm::MemorySSA::VerificationLevel) const: Assertion `&*ALI == *AAI && "Not the same accesses in the same order"' failed.

[22631] signal (6.-6): Aborted
in expression starting at none:0
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f8d572d771a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZNK4llvm9MemorySSA34verifyOrderingDominationAndDefUsesERNS_8FunctionENS0_17VerificationLevelE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZNK4llvm9MemorySSA15verifyMemorySSAENS0_17VerificationLevelE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/src/llvm-julia-licm.cpp:412
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE13runSinglePassIS1_St10unique_ptrINS_6detail11PassConceptIS1_S5_JS4_S7_EEESt14default_deleteISD_EEEENS_8OptionalINS_17PreservedAnalysesEEERT_RT0_RS5_S4_S7_RNS_19PassInstrumentationE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE24runWithoutLoopNestPassesERS1_RS5_S4_S7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE3runERS1_RS5_S4_S7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm25FunctionToLoopPassAdaptor3runERNS_8FunctionERNS_15AnalysisManagerIS1_JEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/src/pipeline.cpp:665
operator() at /home/topolarity/repos/julia/src/jitlayers.cpp:1218
withModuleDo<(lambda at /home/topolarity/repos/julia/src/jitlayers.cpp:1183:30)> at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136
operator() at /home/topolarity/repos/julia/src/jitlayers.cpp:1183
CallImpl<(anonymous namespace)::OptimizerT> at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/ADT/FunctionExtras.h:222
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
emit at /home/topolarity/repos/julia/src/jitlayers.cpp:690
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession12dispatchTaskESt10unique_ptrINS0_4TaskESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
addModule at /home/topolarity/repos/julia/src/jitlayers.cpp:1497
jl_add_to_ee at /home/topolarity/repos/julia/src/jitlayers.cpp:1934
jl_add_to_ee at /home/topolarity/repos/julia/src/jitlayers.cpp:1913
_jl_compile_codeinst at /home/topolarity/repos/julia/src/jitlayers.cpp:243
jl_generate_fptr_impl at /home/topolarity/repos/julia/src/jitlayers.cpp:493
jl_compile_method_internal at /home/topolarity/repos/julia/src/gf.c:2475
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2881
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3ac0b356)
unknown function (ip: 0x7f8d3abea45f)
unknown function (ip: 0x7f8d3abe8500)
unknown function (ip: 0x7f8d3abe62b4)
jl_fptr_args at /home/topolarity/repos/julia/src/gf.c:2531
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
jl_apply at /home/topolarity/repos/julia/src/julia.h:1963
jl_f__call_latest at /home/topolarity/repos/julia/src/builtins.c:812
unknown function (ip: 0x7f8d3ac40266)
unknown function (ip: 0x7f8d3ac40357)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3d4af0db)
unknown function (ip: 0x7f8d3d8c3448)
unknown function (ip: 0x7f8d3d8c42c7)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3d8f2d57)
unknown function (ip: 0x7f8d3d8f00d7)
unknown function (ip: 0x7f8d3d8f00ef)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
jl_apply at /home/topolarity/repos/julia/src/julia.h:1963
_start_task at /home/topolarity/repos/julia/src/task.c:1238
Allocations: 2897347 (Pool: 0; Other: 2897347); GC: 10
Allocations: 2897347 (Pool: 0; Other: 2897347); GC: 10
[1]    22631 IOT instruction  ./tmp/test-asan/asan/julia
@topolarity
Copy link
Member Author

Running with JL_FORCE_JITLINK without ASAN does not reproduce this problem - it appears to be related to some other difference when running with ASAN enabled.

@vchuravy
Copy link
Member

The duplicated section seems similar to #50044

@lhames have you tried JITLink and the sanitizers?

@lhames
Copy link

lhames commented Jun 15, 2023

@vchuravy They didn't work last time I tried (a couple of years back). They went on my unwritten to-do list, but I haven't gotten around to them yet.

I think this is worth opening an LLVM issue for -- at least one other person has asked about it recently.

@topolarity topolarity mentioned this issue Jun 20, 2023
Merged
@pchintalapudi pchintalapudi mentioned this issue Jun 27, 2023
Merged
@vtjnash vtjnash closed this as completed Jun 29, 2023
@maleadt maleadt mentioned this issue Oct 18, 2023
Merged
vtjnash added a commit that referenced this issue Nov 20, 2023
@maleadt @vtjnash
build: ASAN fixes for glibc (#51755)
5cb0e51 
For the `sigsetjmp` bypass; looks like glibc removed the
`__libc_siglongjmp` symbol in glibc 2.34, so change to using the
approach taking by our `dlopen` wrapper instead.

Adopts topolarity's fixes from #50170
Resolves #47698

Co-authored-by: Jameson Nash <[email protected]>
mkitti pushed a commit to mkitti/julia that referenced this issue Dec 9, 2023
@maleadt @vtjnash @mkitti
build: ASAN fixes for glibc (JuliaLang#51755)
4f39714 
For the `sigsetjmp` bypass; looks like glibc removed the
`__libc_siglongjmp` symbol in glibc 2.34, so change to using the
approach taking by our `dlopen` wrapper instead.

Adopts topolarity's fixes from JuliaLang#50170
Resolves JuliaLang#47698

Co-authored-by: Jameson Nash <[email protected]>
KristofferC pushed a commit that referenced this issue Oct 7, 2024
@maleadt @KristofferC
build: ASAN fixes for glibc (#51755)
f3257a0 
For the `sigsetjmp` bypass; looks like glibc removed the
`__libc_siglongjmp` symbol in glibc 2.34, so change to using the
approach taking by our `dlopen` wrapper instead.

Adopts topolarity's fixes from #50170
Resolves #47698

Co-authored-by: Jameson Nash <[email protected]>
(cherry picked from commit 5cb0e51)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lhames @vchuravy @vtjnash @topolarity