Unable to link Pony programs with gcc defaulting to PIE

[Theodus]

Originally reported by james on the user group here

ponyc fails when gcc is used as the linker with the following error:

Writing ./stdlib.o
Linking ./stdlib
/usr/bin/ld.gold: error: ./stdlib.o: requires unsupported dynamic reloc 11; recompile with -fPIC
/usr/bin/ld.gold: error: ./stdlib.o: requires dynamic R_X86_64_32 reloc against 'pony_personality_v0' which may overflow at runtime; recompile with -fPIC
collect2: error: ld returned 1 exit status
Error:
unable to link: gcc -o ./stdlib -O3 -march=native -mcx16 -fuse-ld=gold  ./stdlib.o -L"/home/theodus/dev/ponyc/build/release/" -Wl,-rpath,"/home/theodus/dev/ponyc/build/release/" -L"/home/theodus/dev/ponyc/build/release/../../packages" -Wl,-rpath,"/home/theodus/dev/ponyc/build/release/../../packages" -L"/usr/local/lib" -Wl,-rpath,"/usr/local/lib" -Wl,--start-group -l"rt" -l"crypto" -l"pcre2-8" -l"ssl" -Wl,--end-group  -lponyrt -lpthread -ldl   -lm
Makefile:653: recipe for target 'test' failed
make: *** [test] Error 255

I believe the problem is that stdlib.o is being linked with libponyrt-pic.a which is position-independent, so stdlib.o also has to be made position independent by compiling with --pic. The question is why this doesn't occur when clang is used as the linker.

[SeanTAllen]

@Praetonus I see you did something back in October with Linux and libponyrt-pic. Any thoughts on this?

[Praetonus]

I'm a bit lost on that one. The correct libponyrt without position-independent code is linked so I don't really understand what's going on. It works for me with GCC on Archlinux, maybe it's an issue with the build of GCC used in Ubuntu? Is somebody able to test this on other distributions?

[SeanTAllen]

@Theodus is emailing the user list to try and get more input on "is this related specifically to this version of Ubuntu".

[dipinhora]

This is likely because Ubuntu 16.10 enabled PIE by default for GCC (see: https://wiki.ubuntu.com/SecurityTeam/PIE) which makes it so that the normal libponyrt is also position-independent requiring stdlib.o to also be compiled with --pic.

[sylvanc]

Nice catch, @dipinhora ! I think the options are:

1. Tell users to compile with --pic if their OS prefers it. This is kind of crappy, as it demands that users understand a platform/distro choice that is complex and they shouldn't need to know.
2. Always compile with --pic on Linux. This solves the problem, but at a slight performance cost, since PIC code can have indirections that otherwise wouldn't occur.
3. Always compile with --pic on Linux, but use the bitcode version of libponyrt. Would that work?
4. Somehow detect when a platform/distro has chose PIC by default and use --pic. Not sure how to detect that.
5. Somehow detect as in the above, but when we do, set -no-pie when linking to avoid the PIC performance cost.

Can anybody think of a nice way to detect if -pie is the default?

[Theodus]

We can check for output from the command gcc -v (or gcc -###) and check if the output contains the string "--enable-default-pie".

[Theodus]

During the last sync meeting we decided to experiment with using LLD as the linker rather than depending on whatever linker is used by the C compiler building pony.

[Theodus]

I have finally figured out the combination of flags required to link stdlib.o with LLD, but now the resulting binary results in a segfault. here is the backtrace from LLDB:

(lldb) target create "stdlib"
Current executable set to 'stdlib' (x86_64).
(lldb) run
Process 10273 launched: '/home/theodus/dev/ponyc/stdlib' (x86_64)
Process 10273 stopped
* thread #1: tid = 10273, 0x000000000065a0a0 stdlib`ponyint_opt_next + 144, name = 'stdlib', stop reason = signal SIGSEGV: invalid address (fault address: 0x8)
    frame #0: 0x000000000065a0a0 stdlib`ponyint_opt_next + 144
stdlib`ponyint_opt_next:
->  0x65a0a0 <+144>: movq   (%rsi), %rbp
    0x65a0a3 <+147>: cmpb   $0x2d, (%rbp)
    0x65a0a7 <+151>: jne    0x65a0af                  ; <+159>
    0x65a0a9 <+153>: cmpb   $0x0, 0x1(%rbp)
(lldb) bt
* thread #1: tid = 10273, 0x000000000065a0a0 stdlib`ponyint_opt_next + 144, name = 'stdlib', stop reason = signal SIGSEGV: invalid address (fault address: 0x8)
  * frame #0: 0x000000000065a0a0 stdlib`ponyint_opt_next + 144
    frame #1: 0x0000000000659e0b stdlib`pony_init + 155
    frame #2: 0x0000000000650ad3 stdlib`main + 19

This is the full command used to create the binary using LLD 5.0.0 (trunk 293818):

./build/debug/ponyc -d -r=obj packages/stdlib && \
  ld.lld -o ./stdlib -O3 ./stdlib.o \
  -L/home/theodus/dev/ponyc/build/release/ \
  --rpath /home/theodus/dev/ponyc/build/release \
  -L/home/theodus/dev/ponyc/build/release/../../packages \
  --rpath /home/theodus/dev/ponyc/build/release/../../packages \
  -L/usr/lib/x86_64-linux-gnu/ \
  -L/usr/lib/gcc/x86_64-linux-gnu/6.2.0/ \
  -L/usr/local/lib \
  --rpath /usr/local/lib \
  --start-group -lrt -lcrypto -lpcre2-8 -lssl \
  --end-group -lponyrt -lpthread -ldl -lc -lm -lgcc -lgcc_eh \
  --entry main

[Praetonus]

@Theodus Can you try it with a debug build of the runtime to see if you get any additional information?

[Theodus]

@Praetonus Here is the LLDB output for the binary with the debug runtime:

(lldb) target create "stdlib"
Current executable set to 'stdlib' (x86_64).
(lldb) run
Process 8577 launched: '/home/theodus/dev/ponyc/stdlib' (x86_64)
Process 8577 stopped
* thread #1: tid = 8577, 0x0000000000000000, name = 'stdlib', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
(lldb) bt
* thread #1: tid = 8577, 0x0000000000000000, name = 'stdlib', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000000000000

[Theodus]

I have also tested this with compiler-rt and libunwind instead of libgcc with the same result. Here is the updated command:

./build/debug/ponyc -d -r=obj packages/stdlib && \
  ld.lld-4.0 -o ./stdlib -O3 ./stdlib.o \
  -L/home/theodus/dev/ponyc/build/debug/ \
  --rpath /home/theodus/dev/ponyc/build/debug \
  -L/home/theodus/dev/ponyc/build/debug/../../packages \
  --rpath /home/theodus/dev/ponyc/build/debug/../../packages \
  -L/usr/lib/x86_64-linux-gnu/ \
  -L/usr/lib/llvm-3.9/lib/clang/3.9.1/lib/linux/ \
  -L/usr/local/lib \
  --rpath /usr/local/lib \
  --start-group -lrt -lcrypto -lpcre2-8 -lssl \
  --end-group -lponyrt -lpthread -ldl -lc -lm \
  -lclang_rt.builtins-x86_64 -lunwind \
  --entry=main

[Theodus]

It should also be noted that linking with clang and -fuse-ld=lld-4.0 does not produce this issue.

[jemc]

Had another user report this same problem in #1588. Closed that ticket in favor of this one, and directed this user to this ticket.

[jemc]

Also note that if we solve this issue in an automated way, we can revert #1152, which added instructions telling Arch Linux users to always specify the --pic flag explicitly when compiling Pony programs.

[insanitybit]

I'm on Ubuntu 16.10 and I get this error even when I pass the --pic flag. Also, on a semi-related note, I would be pro "always provide --pic". This is the sane, safe default.

Here is the full error output: http://pastebin.com/G8PQKPZV

Update: I think I had an old version of pony installed + the new version. After removing the old version the --pic flag fix works. I would recommend this as the default.

[SeanTAllen]

Closing in favor of #1811

[mfelsche]

I am reopening this one, as it hits us again on ubuntu 18.04 when compiling with gcc which has --enable-default-pie and LLVM 7.0.1.

Interestingly this only hits us with LLVM 7.0.1 compiled from source (with vendored LLVM using make -f Makefile-lib-llvm ...) or downloaded as pre-compiled libs from releases.llvm.org. The ubuntu-packaged LLVM 7.0.1 works fine.

Steps to reproduce:

• build this Dockerfile with docker build --force-rm -t pony:pie-issue .
• It will fail to compile the helloworld example

FROM "ubuntu:bionic"

ENV LLVM_VERSION=7.0.1
RUN apt update && apt install -y build-essential git zlib1g-dev libncurses5-dev libssl-dev libpcre2-dev xz-utils wget
RUN mkdir llvm && \
    cd /llvm && \
    wget releases.llvm.org/$LLVM_VERSION/clang+llvm-$LLVM_VERSION-x86_64-linux-gnu-ubuntu-18.04.tar.xz && \
    xz -d clang+llvm-$LLVM_VERSION-x86_64-linux-gnu-ubuntu-18.04.tar.xz && \
    ls -ahl && \
    tar xvf clang+llvm-$LLVM_VERSION-x86_64-linux-gnu-ubuntu-18.04.tar && \
    mv clang+llvm-$LLVM_VERSION-x86_64-linux-gnu-ubuntu-18.04 llvm-$LLVM_VERSION
RUN git clone https://github.com/ponylang/ponyc.git && \
    cd ponyc/ && \
    make default_pic=true default_ssl=openssl_1.1.0 LLVM_CONFIG=/llvm/llvm-$LLVM_VERSION/bin/llvm-config && \
    ./build/release/ponyc examples/helloworld/

Failure:

/usr/bin/ld.gold: error: ./helloworld.o: requires dynamic R_X86_64_32 reloc against 'ponyint_personality_v0' which may overflow at runtime; recompile with -fPIC
collect2: error: ld returned 1 exit status

ld.gold is called from cc with -pie which is raising that error. If -pie is removed from the ld.gold call, the link-step succeeds.

Somehow ponyint_personality_v0 is the only offending symbol it seems that does not play well with -pie. In an object file produced by ponyc compiling the helloworld example, the relocations for ponyint_personality_v0 are listed as follows:

Relocation section '.rela.eh_frame' at offset 0x5a60 contains 13 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000020  000300000002 R_X86_64_PC32     0000000000000000 .text + 0
00000000005c  000300000002 R_X86_64_PC32     0000000000000000 .text + 1f0
0000000000bc  000300000002 R_X86_64_PC32     0000000000000000 .text + 490
0000000000f3  004b0000000a R_X86_64_32       0000000000000000 ponyint_personality_v0 + 0
000000000108  000300000002 R_X86_64_PC32     0000000000000000 .text + 8d0
000000000111  00040000000a R_X86_64_32       0000000000000000 .gcc_except_table + 0
000000000158  000300000002 R_X86_64_PC32     0000000000000000 .text + b60
000000000174  000300000002 R_X86_64_PC32     0000000000000000 .text + e20
0000000001c0  000300000002 R_X86_64_PC32     0000000000000000 .text + 13c0
0000000001dc  000300000002 R_X86_64_PC32     0000000000000000 .text + 14e0
000000000218  000300000002 R_X86_64_PC32     0000000000000000 .text + 1770
000000000234  000300000002 R_X86_64_PC32     0000000000000000 .text + 1850
000000000260  000300000002 R_X86_64_PC32     0000000000000000 .text + 1970

In an object file produced by ponyc compiled against LLVM 7.0.1 from apt.llvm.org (here linking succeeds even with -pie) the relocations are listed like this:

Relocation section '.rela.data.DW.ref.ponyint_personality_v0' at offset 0x5a30 contains 1 entry:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000000  004c00000001 R_X86_64_64       0000000000000000 ponyint_personality_v0 + 0

Relocation section '.rela.eh_frame' at offset 0x5a48 contains 13 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000020  000400000002 R_X86_64_PC32     0000000000000000 .text + 0
00000000005c  000400000002 R_X86_64_PC32     0000000000000000 .text + 1f0
0000000000bc  000400000002 R_X86_64_PC32     0000000000000000 .text + 470
0000000000f3  001600000002 R_X86_64_PC32     0000000000000000 DW.ref.ponyint_persona + 0
000000000108  000400000002 R_X86_64_PC32     0000000000000000 .text + 8a0
000000000111  000500000002 R_X86_64_PC32     0000000000000000 .gcc_except_table + 0
000000000158  000400000002 R_X86_64_PC32     0000000000000000 .text + b30
000000000174  000400000002 R_X86_64_PC32     0000000000000000 .text + de0
0000000001c0  000400000002 R_X86_64_PC32     0000000000000000 .text + 1370
0000000001dc  000400000002 R_X86_64_PC32     0000000000000000 .text + 1490
000000000218  000400000002 R_X86_64_PC32     0000000000000000 .text + 1720
000000000234  000400000002 R_X86_64_PC32     0000000000000000 .text + 1800
000000000260  000400000002 R_X86_64_PC32     0000000000000000 .text + 1910

I am not sure where the differences in the generated object files come from.
The LLVM from apt lists the host-target as: x86_64-pc-linux-gnu. vendored and precompiled LLVM both list the target as x86_64-unknown-linux-gnu. Maybe this results in the differences in the object files.

Will investigate further, but marking this as needs discussion during sync as maybe more eyes on this can help.

What might work (as we don't use lld yet) is to detect a pie compiler and issue -nopie during linking. But this would need to be a check during runtime of ponyc.

[SeanTAllen]

Im unable to reproduce this using vendored LLVM from source. ONLY from the 7.0.1 download from LLVM. Are you sure it isn't a problem with how that 7.0.1 download was built?

[SeanTAllen]

To add color @mfelsche, I have built using the vendored LLVM 7.0.1 multiple times without issue.

[mfelsche]

To add even more color: I have two ponycs here, one with vendored LLVM which does work, one with the one from releases.llvm.org which does not work. Both produce identical IR. Bitcode, ASM and object files differ. Fun times!

**EDIT: Mistakes were made. IR is identical, bitcode, asm and object file differs.

[SeanTAllen]

I don't think this issue is the one you've reopened @mfelsche. I have been unable to reproduce except with the LLVM 7.0.1 from LLVM releases. The fix for the previous problem is being done. I think what you are looking at is a different issues.