bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems

[bergkvist]

Improve binary portability between MacOS versions

https://bugs.python.org/issue44689

When compiling CPython using a MacOS Catalina build server, it won't work as expected when trying to run it on MacOS Big Sur. In particular, ctypes.util.find_library(name) will always return None. This PR adds support for compiling on an older MacOS version.

Background

Checking if a shared library exists on MacOS Big Sur is no longer possible by looking at the file system. Instead, Apple recommends the use of dlopen to check if a shared library exists (MacOS Big Sur 11.0.1 changenotes). Note that using dlopen to check for library existence is not ideal, since this might cause arbitrary side effects.

_dyld_shared_cache_contains_path and build time requirements

The current solution (introduced in #22855) to make find_library work on MacOS Big Sur does not use dlopen, but rather _dyld_shared_cache_contains_path from #include <mach-io/dyld>. This function/symbol is only available on MacOS Big Sur, meaning it will only be included if you compile CPython on MacOS Big Sur.

In other words, if we compile CPython on MacOS Catalina, and move the binary to MacOS Big Sur, from _ctypes import _dyld_shared_cache_contains_path will raise an ImportError - and find_library will not work, always returning None.

# ctypes/macholib/dyld.py
# ...

+try:
+    from _ctypes import _dyld_shared_cache_contains_path
+except ImportError:
+    def _dyld_shared_cache_contains_path(*args):
+        raise NotImplementedError

# ...

def dyld_find(name, executable_path=None, env=None):
    """
    Find a library or framework using dyld semantics
    """
    for path in dyld_image_suffix_search(chain(
                dyld_override_search(name, env),
                dyld_executable_path_search(name, executable_path),
                dyld_default_search(name, env),
            ), env):

        if os.path.isfile(path):
            return path
+        try:
+            if _dyld_shared_cache_contains_path(path):
+                return path
+        except NotImplementedError:
+            pass
# ...

Method/Verification

In order to support building for MacOS >= 11.0 (Big Sur or later) with a build server running MacOS < 11.0 (Catalina or earlier), we use dynamic loading to avoid errors when compiling. _dyld_shared_cache_contains_path will be resolved at runtime instead of compile-time, since it is not available at compile-time in this case.

The downside to doing this at runtime instead of compile-time is that the compiler won't give us useful error messages. This is why we want to keep using the weak linking approach when compiling on MacOS >= 11.0 (Big Sur or later) - and only use dynamic loading to support the deprecated use case of compiling on an older MacOS version.

To test that this works, I've set up 2 virtual machines using OSX-KVM on a Manjaro Linux host:

• Catalina (OS: macOS Catalina 10.15.7 19H15 x86_64, Kernel 19.6.0)
• Big Sur (OS: macOS 11.4 20F71 x86_64, Kernel: 20.5.0)

Files are copied between the 2 virtual machines using rsync:

#!/usr/bin/env bash
rm -rf "./local"
echo "Pulling from Catalina... (/usr/local/ -> ./local)"
rsync -a -e "ssh -p 2222" tobias@localhost:/usr/local .
echo "Pushing to BigSur... (./local/ -> /usr/local)"
rsync -a -e "ssh -p 3222" ./local/* tobias@localhost:/usr/local/
echo "Done!"

Checks

• Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c')

Results

Before this change:

❌ Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c')

Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
None

After this change:

✔️ Compile on Catalina and install, copy /usr/local/* over to BigSur, and call find_library('c')

Python 3.11.0a0 (heads/macos-ctypes-find-library:0858c0c9ff, Jul 19 2021, 17:52:24) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
/usr/lib/libc.dylib

Related issues/Symptoms of the issue

Note that some of these are closed due to the symptom having been treated - rather than the underlying cause.

• Python's ctypes broken on macOS Big Sur NixOS/nixpkgs#105038
• Getting AttributeError: dlsym(RTLD_DEFAULT, GetMacOSStatusCommentString): symbol not found Error On MacOS Big Sur jupyterlab/jupyterlab#9863
• MacOS Big Sur "dlsym(RTLD_DEFAULT, GetMacOSStatusCommentString): symbol not found" arsenetar/send2trash#51
• Regarding of Jupyter lab  minrk/appnope#12
• JLab on Big Sur -- AssertionError: Couldn't find Class NSProcessInfo jupyterlab/jupyterlab#9410
• NotImplementedError: _dyld_shared_cache_contains_path not available hbldh/bleak#372
• https://bugs.python.org/issue43052
• https://stackoverflow.com/questions/65065482/error-when-importing-pyautogui-on-macos-10-13-and-python-3-8

https://bugs.python.org/issue44689

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@bergkvist

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[bergkvist]

Since this PR suggests changes to code that was merged in #22855, I'd love a review from @ronaldoussoren
This might also be relevant to @ned-deily, @serhiy-storchaka, @vstinner - since you guys were involved with that PR.

[erlend-aasland]

I'll leave the technical review to Ronald, but I've left some remarks regarding code style (PEP 7).

BTW, did you read the discussion in GH-21241 regarding _dyld_shared_cache_contains_path?

[bergkvist]

Thanks a lot for the feedback @erlend-aasland - and for linking the discussion about _dyld_shared_cache_contains_path. Do you know if there a VSCode plugin for checking or auto-formatting C code according to PEP 7?

I discussed a bit with Ronald on the issue linked to this PR, and he made me realize that using dlopen to check for library existence is not a good idea because it can have side effects. A shared library can execute arbitrary code when opened with dlopen. This is why _dyld_shared_cache_contains_path is currently being used instead.

Instead of checking for _dyld_shared_cache_contains_path at compile time - it is possible to resolve it at run-time (dynamic loading) - and provide a fallback function which always returns false if the symbol does not exist. Then it won't matter whether the compilation happened on Catalina or Big Sur. I'll make an attempt at implementing this over the next few days.

[erlend-aasland]

Do you know if there a VSCode plugin for checking or auto-formatting C code according to PEP 7?

Sorry, I do not; I don't use VSCode, I use Vim.

[yaitskov]

I would like to see this merged too, because having custom build of cpython in nix is a bit pain - some functions are private and has to be copy-pasted.

[ned-deily]

Thanks for the updated PR. Please see my recent detailed response on the bug tracker issue for comments that apply to this version.

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[bergkvist]

I have made the requested changes; please review again

[bedevere-bot]

Thanks for making the requested changes!

@ned-deily: please review the changes made to this pull request.

[ronaldoussoren]

The PR looks good to me, from staring at the code. I have not yet tried building on an older macOS version and running tests on Big Sur.

[ned-deily]

I will review and test it next week.

[miss-islington]

Thanks @bergkvist for the PR, and @ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9, 3.10.
🐍🍒⛏🤖

[bedevere-bot]

@ned-deily: Please replace # with GH- in the commit message next time. Thanks!

[bedevere-bot]

GH-28052 is a backport of this pull request to the 3.10 branch.

[bedevere-bot]

GH-28053 is a backport of this pull request to the 3.9 branch.

[bedevere-bot]

GH-28054 is a backport of this pull request to the 3.8 branch.