Error "ld: library not found for -lSystem" on macOS Big Sur

[lutzh]

Hi, I'm trying to compile a "Hello World" program on macOs Big Sur 11.0.1 (20B29)

$ ponyc --version
0.38.1-bba16525 [release]
Compiled with: LLVM 9.0.1 -- AppleClang-11.0.3.11030032-x86_64
Defaults: pic=true

but linking fails

Writing ./helloworld.o
Linking ./helloworld
ld: library not found for -lSystem
Error:
unable to link: ld -execute -no_pie -arch x86_64 -macosx_version_min 10.12 -o ./helloworld ./helloworld.o -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/" -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/../lib/x86-64" -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/../packages" -L"/usr/local/lib"  -lponyrt -lSystem

Sorry if this is an FAQ... I literally tried pony for the very first time today, so please bear with me...

Thanks for your help!

[SeanTAllen]

Some background that might be helpful (or perhaps not):

• Did you upgrade or do a fresh install of Big Sur?
• Can you give the output of clang -v and which clang?
• How did you install pony?

[SeanTAllen]

Can you give the output of clang -v and which clang as well?

[lutzh]

Hi @SeanTAllen , thanks for looking into this!

• Did you upgrade or do a fresh install of Big Sur?

Upgrade

• Can you give the output of clang -v and which clang?

$ which clang
/usr/bin/clang

and

$ clang -v
Apple clang version 12.0.0 (clang-1200.0.32.27)
Target: x86_64-apple-darwin20.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

• How did you install pony?

I followed the instructions here: https://github.com/ponylang/ponyc/blob/master/INSTALL.md#macos
i.e. I installed ponyup with the provided script and then used that to install ponyc

[SeanTAllen]

Thanks. As far as I can tell (I don't have a Mac and our CI doesn't have Big Sur yet), with Big Sur they moved the location of a number of libraries. That MIGHT have an impact. I'm unsure. I'm trying to find someone in the Pony community who has a Big Sur install who might be able to help diagnose more fully. The limited information that I've found online for people experiencing similar problems with non-pony stuff is rather scattershot and doesn't provide any solid "it's definitely X" answers.

[SeanTAllen]

@lutzh can you run sudo xcode-select --install to make sure you have all the latest xcode command line tools installed?

[lutzh]

I did, I don't think that's the problem, the output is

$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

but I'm pretty sure the XCode command line tools and everything else are up to date, there are no updates displayed in "Software Update".

I'm not quite sure where/how ld is invoked. Is there a way to pass additional arguments for the library path to the ponyc compiler?

[SeanTAllen]

Thanks @lutzh.

It does indeed look up to date.

There's a --linker option that you can use but you need to supply the entire "ld" command not just additions.

This is the command that is failing. You should be able to try changing to it as long after it fails once you don't run any other ponyc commands in the same directory (at least this is the command based on what you pasted- assuming it is correct and there was no copy/paste etc munging):

ld -execute -no_pie -arch x86_64 -macosx_version_min 10.12 -o ./helloworld ./helloworld.o -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/" -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/../lib/x86-64" -L"[..]/ponyup/ponyc-release-0.38.1-x86_64-darwin/bin/../packages" -L"/usr/local/lib"  -lponyrt -lSystem

It's quite possible that an additional path is needed for a -L to indicate where to find the System library. That in my mind is the most likely issue. If you can figure it out, we can do a PR to update the MacOS version going forward.

Here's the code in ponyc where the ld command is set up:

ponyc/src/libponyc/codegen/genexe.c

Line 295 in 7c05e1f

snprintf(ld_cmd, ld_len,

[lutzh]

Hi @SeanTAllen ,

Thanks again. Tried that, I can confirm that

ponyc --linker="ld -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib"

does work and I got to the the "Hello, world!" output I was looking for.

So it might be that some lookup rules changed in macOS 10.12 and this has to be passed to the linker as a library path. Or it might still be that some configuration is messed up just on my machine. I guess I could create a PR, but I hope what fixes it for me doesn't break it for anyone else. Do you really think I should?

[SeanTAllen]

From what I've read, this appears to be something that will be needed in general for Big Sur.

I'm surprised the xcode update didn't pick that change up automatically as what I read seemed to indicate that it would.

Can you run "which ld" and put the output here.

There's a couple other things to check which I'll add another note for once I'm not on my phone.

[SeanTAllen]

@lutzh what happens if you run:

ponyc --linker="ld -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -lSystem"

Do you get a working hello world application?

[lutzh]

@lutzh what happens if you run:

ponyc --linker="ld -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -lSystem"

Do you get a working hello world application?

Yes I did, see my comment #3684 (comment)
Well. I did not put in the "-lSystem" as that seemed redundant, but a quick check revealed it would still work if I did.

As for your other question,

$ which ld
/usr/bin/ld

Maybe also of interest

$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-609.7
BUILD 18:10:07 Oct 19 2020
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 12.0.0, (clang-1200.0.32.27) (static support for 27, runtime is 27)
TAPI support using: Apple TAPI version 12.0.0 (tapi-1200.0.23.4)

[SeanTAllen]

Excellent. Thanks. So it appears the solution would be to add -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib when on Big Sur or higher. (I believe that is version 11.0).

In link_exe we don't currently distinguish between different Mac versions when doing any of the work. I think therefore, it would reasonable to add the -L addition in general when on MacOS as I don't think that it causes any issues (but I'm not sure- CI should catch if it does).

The primary change therefore can be done here in genexe.c:

ponyc/src/libponyc/codegen/genexe.c

Line 295 in 7c05e1f

snprintf(ld_cmd, ld_len,

The only caveat is making sure that the memory to be allocated is adjusted at

ponyc/src/libponyc/codegen/genexe.c

Line 290 in 7c05e1f

size_t ld_len = 128 + arch_len + strlen(linker) + strlen(file_exe) +

[SeanTAllen]

@lutzh this is ready for someone to pick up the work. Would you like to do it? It's straightforward. You did most of the diagnosing, so I'd like to offer you the opportunity to fix it if you are interested, if you aren't that's ok too.

[lutzh]

Tbh given I only tried out ponyc for the first time this weekend and have never built it from source, it might be better someone else takes over from here. When I explore pony further, I'm sure there'll be ample opportunity to find some other issue to work on.

[v-almonacid]

I just ran into this github issue since I'm facing a similar issue (can't compile an app that uses the ld linker after upgrading to Big Sur).

Just wanted to point out that I'm on a recently upgraded Big Sur machine and this directory doesn't exist anymore:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/

[SeanTAllen]

@v-almonacid does this mean that the fix that worked for @lutzh doesn't work for you?

[v-almonacid]

@SeanTAllen I don't know, I'm not using this software. It's just a heads up.

[lutzh]

@v-almonacid Apple apparently had the ingenuous idea to delete the command line tools with every XCode update, but you should be able to make them reappear with xcode-select --install.
At least that brought them (and thus /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib/) back for me after the last update. Have you tried that?

[v-almonacid]

@lutzh I did an upgrade to the command line tools just last night (this is a ~month after I upgraded to Big Sur), and even after doing xcode-select install I don't see anything in that path.

I wonder whether this is related to (from Big Sur change logs):

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

[v-almonacid]

@lutzh sorry, forget what I say, I think it's just that the upgrade was not complete (needed to accept license agreement, etc).
Apologies for polluting this thread 😬

[ergl]

I'm going to give this a try this weekend, so I'm assigning it to myself for now.

[alikhaledi]

I am using ifort on Big Sur and I ran into this issue, I have read through the comments but still, I am not sure how to resolve the issue. Can someone help me, please.

[ergl]

Hi @alikhaledi the problem should be fixed in the next release. In the meantime, you can try compiling your programs like so:

ponyc --linker="ld -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -lSystem"

Let us know if that works for you!

[SeanTAllen]

This is fixed in 0.38.2 which was released within the last 10 minutes.

[alikhaledi]

Yes, I did but still, I get the same error.

I have the following simple FORTRAN code

Implicit none

print*, 'Hello'

end

Hi @alikhaledi the problem should be fixed in the next release. In the meantime, you can try compiling your programs like so:

ponyc --linker="ld -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -lSystem"

Let us know if that works for you!

[SeanTAllen]

@alikhaledi this isn't a fortran project. you'll need to consult with the project in question. this is an issue for the Pony programming language compiler only.