PyObjC built with Xcode 15 RC fails to load on macOS 10.13 (new linker bug)

[mrpippy]

Describe the bug
I built PyObjC 9.2 using Xcode 15 RC (15A240d), with MACOSX_DEPLOYMENT_TARGET=10.13, and _objc.so fails to load on macOS 10.13. __dyld_shared_cache_contains_path cannot be found:

Symbol not found: __dyld_shared_cache_contains_path
  Referenced from: /Users/pip/macosx/CrossOver/build/Build/Products/Release/CrossOver.app/Contents/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyobjc_core-9.2-py3.10-macosx-10.13-x86_64.egg/objc/_objc.cpython-310-darwin.so
  Expected in: /usr/lib/libSystem.B.dylib

This function is called inside an @available() block, so it should be weak-linked, but it is not:

$ nm -m -u _objc.cpython-310-darwin.so |grep shared_cache
                 (undefined) external __dyld_shared_cache_contains_path (from libSystem)

(this should be weak external)

This turned out to be a bug with the new linker in Xcode 15 when using ThinLTO. I made a standalone testcase, and the symbol is correctly weak-linked when using the old linker (-Wl,-ld_classic).

Here's the sample file (rename to testdsd.m): testdsd.m.txt

Using the new linker with ThinLTO (which PyObjC does) fails. Old linker+ThinLTO works correctly:

$ MACOSX_DEPLOYMENT_TARGET=10.13 clang -arch x86_64 -dynamic -c -o testdsd.o testdsd.m -flto=thin
$ MACOSX_DEPLOYMENT_TARGET=10.13 clang -arch x86_64 -bundle -o testdsd.dylib testdsd.o
$ nm -m -u testdsd.dylib | grep shared_cache
                 (undefined) external __dyld_shared_cache_contains_path (from libSystem)
$ MACOSX_DEPLOYMENT_TARGET=10.13 clang -arch x86_64 -bundle -o testdsd.dylib testdsd.o -Wl,-ld_classic
$ nm -m -u testdsd.dylib | grep shared_cache
                 (undefined) weak external __dyld_shared_cache_contains_path (from libSystem)

New linker without ThinLTO also works correctly:

$ MACOSX_DEPLOYMENT_TARGET=10.13 clang -arch x86_64 -dynamic -c -o testdsd.o testdsd.m
$ MACOSX_DEPLOYMENT_TARGET=10.13 clang -arch x86_64 -bundle -o testdsd.dylib testdsd.o
$ nm -m -u testdsd.dylib | grep shared_cache
                 (undefined) weak external __dyld_shared_cache_contains_path (from libSystem)

Platform information

Python 3.10, built from source.
Runs correctly on macOS Ventura, fails to launch on High Sierra (and likely also Mojave and Catalina).

Additional context
I filed FB13171424 with Apple for this.

Disabling -flto=thin (using the --no-lto argument to build_ext) as a temporary workaround would probably work, but the linker should really be fixed instead. Until it is, Xcode 14 should be used if building PyObjC for pre-Big Sur OSes.

[ronaldoussoren]

Thanks for reporting this. I don't test wheels on older macOS versions a lot and likely would have missed this issue because of that.

Using Xcode 14 isn't really an option either due to needing to support new APIs in the framework bindings. The upcoming PyObC 10 release needs to be build using Xcode 15 because of that.

I'll probably end up using the old linker, it shouldn't be too hard to automatically detect Xcode 15 and select the old linker for that.

[ronaldoussoren]

BTW. This is a known issue with Xcode 15, although it is downplayed a lot ("this primarily affects C++ projects"). See https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Linking.

[ronaldoussoren]

I have implemented the workaround for PyObjC 10. Will close the issue when I've tested a full build and checked that the resulting wheels work on an older macOS version.

[ronaldoussoren]

Tested a wheel of PyObjC 10 built with Xcode 15 on a 10.11 system and 'import objc' works.

[mrpippy]

Apple replied to my FB:

You can workaround it using "-weak_reference_mismatches weak" linker option, which will instruct linker to prefer weak references in case of a mismatch.
Alternative is to use the "-ld_classic" option to fallback to the old linker until this is resolved.

[mrpippy]

This bug seems to be fixed with Xcode 15.1b2 (testing with my minimal example, haven't tried full PyObjC yet)

[cpatulea]

I think 966ddd0 broke the ability to build using Command-Line Tools only (without full Xcode). Here is what I get:

running build_ext
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
Traceback (most recent call last):
  File "/Users/catalinp/gtk/source/pyobjc-core-10.0/setup.py", line 723, in <module>
    setup(
  File "/Users/catalinp/gtk/inst/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
  File "/Users/catalinp/gtk/inst/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['xcodebuild', '-version']' returned non-zero exit status 1.

It seems in the past there was some intent for pyobjc to build with solely command-line tools: af03496

There are tips here on how to detect version with just command-line tools: #496 [edit] and here: https://github.com/Xpra-org/gtk-osx-build/blob/9c7b9adba36441b1606ef62689d2c0e716c5a9a7/jhbuildrc-gtk-osx#L137