[POWERPC64*] llvm12 and later produces broken binaries on FreeBSD

[adalava]

Bugzilla Link	51590
Version	unspecified
OS	FreeBSD
Blocks	#48661 #50580 #51489
CC	@emaste,@luporl,@pkubaj,@zygoloid,@tstellar

Extended Description

LLVM12 and later (tested main branch at 2c5c06c) produces broken binaries on FreeBSD 13 for powerpc64 and powerpc64le. The binaries will crash with segmentation.

Steps to reproduce using FreeBSD 13 RELEASE ISO:

1. Install llvm12 (apt install llvm12)
2. Create echo_crash.c:

#include <capsicum_helpers.h>
#include <err.h>

int
main(int argc, char *argv[])
{

    if (caph_limit_stdio() < 0 || caph_enter() < 0)
            err(1, "capsicum");
    return 0;

}

3. Compile with:

clang12 -O2 -g echo_crash.c -o echo_crash.good
clang12 -O2 -g -fstack-protector-strong echo_crash.c -o echo_crash.bad

4. Execute the binaries produced.
   ./echo_crash.good: will produce no output and return normally.
   ./echo_crash.bad: Segmentation Fault

[adalava]

assigned to @adalava

[adalava]

The same occurs with clang/LLVM built from sources (tested main branch at 2c5c06c).
When building from sources you should use llvm11, as this is the latest/best working compiler.

[adalava]

During my tests, a working binary is produced when -fstack-protector-strong is removed.
Strangely a working binary is also produced if -O[0|1|2|3] is removed while -fstack-protector-strong is kept

[adalava]

0x10010eb4 <caph_limit_stream> addis r2,r12,2
0x10010eb8 <caph_limit_stream+4> addi r2,r2,-31372
0x10010ebc <caph_limit_stream+8> mflr r0
0x10010ec0 <caph_limit_stream+12> std r27,-40(r1)
0x10010ec4 <caph_limit_stream+16> std r28,-32(r1)
0x10010ec8 <caph_limit_stream+20> std r29,-24(r1)
0x10010ecc <caph_limit_stream+24> std r30,-16(r1)
0x10010ed0 <caph_limit_stream+28> std r0,16(r1)
0x10010ed4 <caph_limit_stream+32> stdu r1,-160(r1)
0x10010ed8 <caph_limit_stream+36> mr r29,r3
0x10010edc <caph_limit_stream+40> li r3,129
0x10010ee0 <caph_limit_stream+44> addis r27,r2,-4099
0x10010ee4 <caph_limit_stream+48> rotldi r28,r3,57

0x10010ee8 <caph_limit_stream+52> ld r3,27608(r27) <--- segmentation fault
0x10010eec <caph_limit_stream+56> mr r30,r4
0x10010ef0 <caph_limit_stream+60> li r4,2049
0x10010ef4 <caph_limit_stream+64> addi r9,r28,11
0x10010ef8 <caph_limit_stream+68> std r3,112(r1)
0x10010efc <caph_limit_stream+72> lis r3,1024
0x10010f00 <caph_limit_stream+76> rotldi r5,r4,58

[adalava]

objdump output with clang11 and clang12
This contains the objdump output of same program compiled with both "-O2 and -fstack-protector-strong" on llvm11 (working) and llvm12(crashing)

[adalava]

1. Install llvm12 (apt install llvm12)

I meant "pkg install llvm12" ;-)

[adalava]

An alternative test program that doesn't rely on FreeBSD libraries is:

int main(int argc, char *argv[])
{
char name[10];
if (ttyname_r(0, name, 10))
err(1, "capsicum"); return 0;
};

build parameters:
-target powerpc64-unknown-freebsd13 -O2 -g -fstack-protector-strong

[adalava]

Bisect points to commit 2518433
A quick workaround is under review on FreeBSD project. Comments are welcome: https://reviews.freebsd.org/D31698

[adalava]

llvm12 FreeBSD in-tree fix
This is the fix/workaround commited to FreeBSD's in-tree LLVM

[tstellar]

Are you going to submit this fix via phabricator?

[adalava]

Are you going to submit this fix via phabricator?

Just submitted it as https://reviews.llvm.org/D109090

I'm not sure who should review it. Feel free to edit it if you wish.

[adalava]

The commit https://reviews.llvm.org/rG1cb9f37a17ab528e53d38c790c1bc44a087d2a50 fixed the problem on FreeBSD/powerpc64, however caused a regression on FreeBSD/amd64 freestanding objects used in kernel.

It seems like there's a problem somewhere else (check discussion on https://reviews.llvm.org/D109090).

To resolve the regression the patch should be amended to restrict the logic to PPC64 only, but more investigation is still required.

[adalava]

To resolve the regression the patch should be amended to restrict the logic
to PPC64 only, but more investigation is still required.

Added https://reviews.llvm.org/D113443

[tstellar]

mentioned in issue #51489