[OpenMP] Clang: cannot compile minimal example to wasm32/wasm64

[mf-RDP]

Hi,

this is a copy of issue 186 in https://github.com/WebAssembly/threads/issues/
Please see comments from sbc100 & sunfishcode there.

Compiling a minimal OpenMP example:

#pragma optimize off
#include <stddef.h>
#include <stdint.h>
#include <stdlib.h>
#include "anet_intrinsic.h"
//#include <omp.h>

int main() {
#pragma omp parallel num_threads(4)
{
#pragma omp critical
abs(-5555);
}
}

fails:
C:\LLVM\BUILDLLDB\Debug\bin\clang++ -fopenmp -DDEBUG -mbulk-memory -mmutable-globals -O0 -I.. -I.... -I. -Wl,--max-memory=327680 -Wl,--shared-memory -matomics -target wasm32-wasi --sysroot /project/wasi-libc/sysroot-intrin-bulk-memory -Wl,--export-all -Wl,-L. -Wl,--entry=__main_void -Wl,--allow-undefined -o omp_dummy.cc.wasm omp_dummy.cc
Common symbols are not yet implemented for Wasm

While - for the moment - maybe we can leave compiling libomp to the embedder or he can provide appropriate _kmp* functions as runtime imports, providing compilation support would be a great step to OpenMP support.

Question would probably be: can be worked around needing common symbols for omp compilation.

Regards,
S.

[dschuff]

From WebAssembly/threads#186

[mf-RDP]

Thanks, Pierre, I confirm the issue (for wasm) narrows down to omp critical.
Maybe there is any chance to eliminate the use of common symbols on this.

Thanks, S.

[shiltian]

@mf-RDP is it still an issue now?

[arsnyder16]

@shiltian @mf-RDP I was able to get omp working with the attached patch to the 14.0.0 branch.

As i am sure someone with more intimate knowledge of the omp library may see some better ways of doing these things

wasm.patch

After apply the patch

emcmake cmake \
  -DOPENMP_STANDALONE_BUILD=ON \
  -DOPENMP_ENABLE_OMPT_TOOLS=OFF \
  -DOPENMP_ENABLE_LIBOMPTARGET=OFF \
  -DLIBOMP_HAVE_OMPT_SUPPORT=OFF \
  -DLIBOMP_OMPT_SUPPORT=OFF \
  -DLIBOMP_OMPD_SUPPORT=OFF \
  -DLIBOMP_USE_DEBUGGER=OFF \
  -DLIBOMP_FORTRAN_MODULES=OFF \
  -DLIBOMP_ENABLE_SHARED=OFF \
  -DLIBOMP_ARCH=wasm32 \
  -DOPENMP_ENABLE_LIBOMPTARGET_PROFILING=OFF \
  .

emmake make -j

Then to downstream libraries must compile with
-pthread -fopenmp=libomp

I have found that some critical variables in the downstream code consuming this custom built wasm .a may need to create their own assembly(.S) to define the user name critical variables. My assumption here was possibly missing backend to clang omitting the correct assembly?

[penzn]

@arsnyder16 can you please upload your patch to https://reviews.llvm.org as a WIP review, this way people would be able to comment on the contents. I would suspect that assembly sources should be part of libomp, but I can be wrong, or maybe it can be either way. Wasm has its own atomic operations for synchronization.

[shiltian]

As @penzn mentioned, those assembly has to be properly added to libomp and when compiling libomp, LIBOMP_ARCH has to be set properly.
@arsnyder16 Could you please upstream it to Phabricator such that it can get a proper review?

[arsnyder16]

@penzn @shiltian Sorry i am not familiar with the process involved here. Someone will need to step me through this. I certainly don't need ownership of this, i am sort of attack the problem for my use case and moved on. I just recently notice this was being discussed so i thought i would offer what i came up.

I am happy to get the ball rolling. I am not familiar with Phabricator or where to upload the patch to https://reviews.llvm.org/

[arsnyder16]

If there are questions about some of the changes in the patch, i am more than happy to explain what i experienced and my interpretation of why the change was needed

[penzn]

@arsnyder16 thank you for the patch anyway! Sorry for the extra steps, it is easier to comment on the change directly than referencing lines and files in comments here.

I am happy to get the ball rolling. I am not familiar with Phabricator or where to upload the patch to https://reviews.llvm.org/

reviews.llvm.org is an LLVM-owned instance of Phabricator, the process is pretty simple if you just use your Github account: login with Github and create a patch (pick LLVM Github Monorepo when it asks for repository), it will let you simply upload a diff. Diff should be ideally regenerated with -U999999 flag, that would show entire files.

Let us know if you have any difficulties, I am sure we can work this out.

[arsnyder16]

@penzn Thanks! https://reviews.llvm.org/D142593

[csegarragonz]

Hi!

I have a similar use-case where we build our own LLVM with -DLLVM_ENABLE_RUNTIMES=openmp targetting WebAssembly, and then provide our own host-side implementation of the kmp symbols we need.

This worked fine with our build from the 13 branch, but when moving to 17.0.6 we start seeing some frontend crashes (i.e. compiling some functions fails, whereas others succeed). Any hints at what could be going on in here?

/usr/local/faasm/toolchain/bin/clang++ --target=wasm32-wasi-threads --sysroot=/usr/local/faasm/llvm-sysroot -D__faasm  -O3 -msimd128 --sysroot=/usr/local/faasm/llvm-sysroot -m32 -DANSI -D__faasm -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_PRO
CESS_CLOCKS -D_WASI_EMULATED_SIGNAL -pthread -fopenmp -g -std=gnu++17 -MD -MT omp/CMakeFiles/default_shared.dir/default_shared.cpp.obj -MF omp/CMakeFiles/default_shared.dir/default_shared.cpp.obj.d -o omp/CMakeFiles/default_shared.dir/defa
ult_shared.cpp.obj -c /code/cpp/func/omp/default_shared.cpp
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /usr/local/faasm/toolchain/bin/clang++ --target=wasm32-wasi-threads --sysroot=/usr/local/faasm/llvm-sysroot -D__faasm -O3 -msimd128 --sysroot=/usr/local/faasm/llvm-sysroot -m32 -DANSI -D__faasm -D_WASI_EMULATED_G
ETPID -D_WASI_EMULATED_PROCESS_CLOCKS -D_WASI_EMULATED_SIGNAL -pthread -fopenmp -g -std=gnu++17 -MD -MT omp/CMakeFiles/default_shared.dir/default_shared.cpp.obj -MF omp/CMakeFiles/default_shared.dir/default_shared.cpp.obj.d -o omp/CMakeFil
es/default_shared.dir/default_shared.cpp.obj -c /code/cpp/func/omp/default_shared.cpp
1.      <eof> parser at end of file
2.      Code generation
 #0 0x000055ffdb7ecbab llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/faasm/toolchain/bin/clang+++0x1366bab)
 #1 0x000055ffdb7eb7c9 llvm::sys::RunSignalHandlers() (/usr/local/faasm/toolchain/bin/clang+++0x13657c9)
 #2 0x000055ffdb7a1851 (anonymous namespace)::CrashRecoveryContextImpl::HandleCrash(int, unsigned long) CrashRecoveryContext.cpp:0:0
 #3 0x000055ffdb7a1bc0 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #4 0x00007fee33643520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #5 0x00007fee336979fc pthread_kill (/lib/x86_64-linux-gnu/libc.so.6+0x969fc)
 #6 0x00007fee33643476 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x42476)
 #7 0x00007fee336297f3 abort (/lib/x86_64-linux-gnu/libc.so.6+0x287f3)
 #8 0x000055ffdaf673cc (anonymous namespace)::AAInstanceInfoReturned::updateImpl(llvm::Attributor&) AttributorAttributes.cpp:0:0
 #9 0x000055ffdbe553a7 llvm::AsmPrinter::doFinalization(llvm::Module&) (/usr/local/faasm/toolchain/bin/clang+++0x19cf3a7)
#10 0x000055ffdb558ee0 llvm::FPPassManager::doFinalization(llvm::Module&) (/usr/local/faasm/toolchain/bin/clang+++0x10d2ee0)
#11 0x000055ffdb55f0c4 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/local/faasm/toolchain/bin/clang+++0x10d90c4)
#12 0x000055ffdb8ed6fd clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::Back
endAction, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream>>) (/usr/local/faasm/toolchain/bin/clang+++0x14676fd)
#13 0x000055ffdc0619ce clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/local/faasm/toolchain/bin/clang+++0x1bdb9ce)
#14 0x000055ffdc79ec74 clang::ParseAST(clang::Sema&, bool, bool) (/usr/local/faasm/toolchain/bin/clang+++0x2318c74)
#15 0x000055ffdbd3d4b9 clang::FrontendAction::Execute() (/usr/local/faasm/toolchain/bin/clang+++0x18b74b9)
#16 0x000055ffdbd00634 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/local/faasm/toolchain/bin/clang+++0x187a634)
#17 0x000055ffdbd98b96 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/local/faasm/toolchain/bin/clang+++0x1912b96)
#18 0x000055ffdb0d4182 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/usr/local/faasm/toolchain/bin/clang+++0xc4e182)
#19 0x000055ffdb0d036c ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#20 0x000055ffdbc36bc1 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*)
 const::'lambda'()>(long) Job.cpp:0:0
#21 0x000055ffdb7a1cde llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/usr/local/faasm/toolchain/bin/clang+++0x131bcde)
#22 0x000055ffdbc3733d clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) Job.cpp:0:0
#23 0x000055ffdbc19884 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/usr/local/faasm/toolchain/bin/clang+++0x1793884)
#24 0x000055ffdbc199a9 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/usr/local/faasm/toolchain/bin/clang+++0x17939a9)
#25 0x000055ffdbc1ed07 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/usr/local/faasm/toolchain/bin/clang+++0x1798d07)
#26 0x000055ffdb0d2a43 clang_main(int, char**, llvm::ToolContext const&) (/usr/local/faasm/toolchain/bin/clang+++0xc4ca43)
#27 0x000055ffdaf6d017 main (/usr/local/faasm/toolchain/bin/clang+++0xae7017)
#28 0x00007fee3362ad90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
#29 0x00007fee3362ae40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#30 0x000055ffdb0cf695 _start (/usr/local/faasm/toolchain/bin/clang+++0xc49695)
clang++: error: clang frontend command failed with exit code 134 (use -v to see invocation)
clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18)
Target: wasm32-unknown-wasi-threads
Thread model: posix
InstalledDir: /usr/local/faasm/toolchain/bin
clang++: note: diagnostic msg:

[dschuff]

The only clue in that stack trace is that the actual crash happened in (anonymous namespace)::AAInstanceInfoReturned::updateImpl(llvm::Attributor&) AttributorAttributes.cpp:0:0 but there's no assertion failure message or anything like that. The thing that would help the most is to include the preprocessed source and script that gets dumped along with the stack trace. But more generally I can't tell whether this is the same issue as OP or not.

[csegarragonz]

FWIW, bumping to 18.1.0-rc2 fixed the pre-processor error.