Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python's ctypes broken on macOS Big Sur #105038

Closed
dhess opened this issue Nov 26, 2020 · 31 comments
Closed

Python's ctypes broken on macOS Big Sur #105038

dhess opened this issue Nov 26, 2020 · 31 comments
Labels
0.kind: bug Something is broken 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: darwin Running or building packages on Darwin 6.topic: python

Comments

@dhess
Copy link
Contributor

dhess commented Nov 26, 2020

Python's ctypes support is broken on macOS Big Sur. This currently affects nixpkgs. Here's an example with python3.8-regex:

https://gist.github.com/dhess/e1b2662a6732618b252096e0de3a04d4

This has been fixed in CPython trunk and backported to 3.9:

python/cpython#22855
python/cpython#23295

Unfortunately, it's not yet been backported to 3.8, which is the default python3Packages set in nixpkgs.

@dhess dhess added the 0.kind: bug Something is broken label Nov 26, 2020
@FRidh FRidh added 6.topic: darwin Running or building packages on Darwin 6.topic: python labels Nov 26, 2020
@FRidh
Copy link
Member

FRidh commented Nov 26, 2020

That's a pretty big backport! Probably in time it will be backported to 3.8 as well.

@jonringer
Copy link
Contributor

once they do another release, these will be added "automatically"

@rb2k
Copy link
Contributor

rb2k commented Feb 8, 2021

Has this actually been resolved in any new release?
I think 3.8.7 as well as 3.9.1 are currently both failing the find_library call for me on Big Sur (fine on Catalina)

# sw_vers
ProductName:	macOS
ProductVersion:	11.2
BuildVersion:	20D64

3.8.7:

nix/store/zlck50w2v1ypwk9yqkfy8mf4kmcs89c9-python3-3.8.7/bin/python3.8
Python 3.8.7 (default, Feb  5 2021, 19:55:34) 
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
>>> 

And 3.9:

nix/store/fk1lwfjvpzzh7d1sj8jyi832kzqczsmi-python3-3.9.1/bin/python3.9
Python 3.9.1 (default, Feb  5 2021, 19:44:28) 
[Clang 7.1.0 (tags/RELEASE_710/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
>>> 

@i077
Copy link
Contributor

i077 commented Feb 9, 2021

Getting a similar error when starting jupyter lab (which uses ctypes) on Big Sur (please forgive the ugly shell invocation):

% nix shell --expr '(builtins.getFlake "github:NixOS/nixpkgs/nixpkgs-unstable").legacyPackages.x86_64-darwin.python39.withPackages (ps: [ ps.jupyterlab ])' --impure -c jupyter-lab
Traceback (most recent call last):
  File "/nix/store/nzw8mizwzxxadhcwxfr1ldis8wabk50x-python3.9-jupyterlab-2.2.9/bin/.jupyter-lab-wrapped", line 6, in <module>
    from jupyterlab.labapp import main
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab/labapp.py", line 14, in <module>
    from jupyterlab_server import slugify, WORKSPACE_EXTENSION
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/__init__.py", line 4, in <module>
    from .app import LabServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/app.py", line 9, in <module>
    from .server import ServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/jupyterlab_server/server.py", line 20, in <module>
    from notebook.notebookapp import aliases, flags, NotebookApp as ServerApp
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/notebook/notebookapp.py", line 86, in <module>
    from .services.contents.filemanager import FileContentsManager
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/notebook/services/contents/filemanager.py", line 17, in <module>
    from send2trash import send2trash
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/send2trash/__init__.py", line 12, in <module>
    from .plat_osx import send2trash
  File "/nix/store/hvqz59hsvpsd7ljy58rr687z55dqfc7l-python3-3.9.1-env/lib/python3.9/site-packages/send2trash/plat_osx.py", line 17, in <module>
    GetMacOSStatusCommentString = Foundation.GetMacOSStatusCommentString
  File "/nix/store/nvxkg44pcrijvqmv5m6inz36rf7zgshy-python3-3.9.1/lib/python3.9/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/nix/store/nvxkg44pcrijvqmv5m6inz36rf7zgshy-python3-3.9.1/lib/python3.9/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, GetMacOSStatusCommentString): symbol not found

My guess is that it has something to do with the patch to ctypes being applied to all the python derivations.

@jonringer
Copy link
Contributor

Has this actually been resolved in any new release?

Upsteam hasn't cut any new releases yet.

@i077
Copy link
Contributor

i077 commented Feb 9, 2021

But cpython 3.9.1 (which is in nixpkgs-unstable) should work on Big Sur since it has the patch from python/cpython#23295, right?

@i077
Copy link
Contributor

i077 commented Feb 9, 2021

For reference, the latest Python 3.9 from Homebrew works on Big Sur

% which python3
/usr/local/bin/python3
% python3      
Python 3.9.1 (default, Jan 27 2021, 11:07:38) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; find_library("c")
'/usr/lib/libc.dylib'
>>> 

@fanzeyi
Copy link
Contributor

fanzeyi commented Feb 16, 2021

It works on Homebrew because they manually applied the patch:

https://github.com/Homebrew/homebrew-core/blob/012fff6f2506fc351fc04b5bb1516669614bcd19/Formula/python%403.9.rb#L78-L86

Nevermind, I was distracted while reading through the Homebrew's commit history, this has been removed in their 3.9.1 update. I'm still not sure why Homebrew's Python 3.9.1 works but not nix

@fanzeyi
Copy link
Contributor

fanzeyi commented Feb 16, 2021

So Python 3.9.1 should have ctypes work on Big Sur. Nix's Python still doesn't work because nixpkg's build doesn't provide mach-o/dyld.h header. Causing this check to fail:

python/cpython@4176193#diff-90d08e583c4c9c6f391b2ae90f819f600a6326928ea9512c9e0c6d98e9f29ac2R12040

I'm checking if there is any way to fix it. (I saw there is nixpkgs.darwin.dyld but it doesn't seem to solve the issue)

@fanzeyi
Copy link
Contributor

fanzeyi commented Feb 16, 2021

Oh well, this is a fun digging.

So apparently dyld in nixpkgs was last updated in macOS 10.12 release to 433.5. The function used in fixing Python's ctypes issue appeared in dyld-832.7.1.tar.gz. So this issue probably is gonna hinge on #101229 (That one is only updating to 10.13, which is still three versions away from having _dyld_shared_cache_contains_path.) Now I can certainly write an overlay locally to update dyld and build Python with it but I don't know if there is a good way to have the Python in nixpkg supports it.

Update: now I realized simply updating dyld will not work since it depends on xnu (because of the Availability.h header), and updating xnu seems to be tricky (heavily patched in Nix and I don't want to rebuild the world). I resorted to patching mach-o/dyld.h with only the addition of the new function (It's just an extern so why not?), and one more patch to Python bypassing a runtime check for reasons I don't exactly understand (possibly related to MACOSX_DEPLOYMENT_TARGET setting).

My messy overlay file and patches (note this is very reckless so probably only use if you know what it means): https://gist.github.com/fanzeyi/3af0396eb5cfaa81ad30af427b1e8069

@groodt
Copy link
Contributor

groodt commented Mar 5, 2021

@FRidh As a maintainer of the cpython package, do you know if there are steps towards a resolution or issues we should be tracking here?

I do of course realise that cpython on Darwin is a challenge. If we have some plans or comms, then perhaps the community can assist in some way.

@FRidh
Copy link
Member

FRidh commented Mar 5, 2021

@groodt I don't work with darwin and I am not aware of any PR's either towards adding support for big sur.

@groodt
Copy link
Contributor

groodt commented Mar 8, 2021

@LnL7 @jonringer @domenkozar Do we have anywhere public where issues such as this are prioritised? It feels like this Python issue is only going to be the start of other major blockers for darwin as more and more packages expect newer versions of Apple SDKs. I fully understand that macOS support for Nix is always a struggle, just perhaps hoping that there is an issue being tracked around Big Sur support. Not to mention M1 Apple Silicon support at some stage.

@domenkozar
Copy link
Member

@groodt As part of https://opencollective.com/nix-macos the goal is also to come up with a central place to track macOS support in a systematic way.

@toonn
Copy link
Contributor

toonn commented Mar 8, 2021

@groodt, #105026 tracks Apple Silicon support. I will be working towards tracking the status of macOS support.

There is a great deal of friction between supporting the broadest possible userbase, which requires a stdenv based on the oldest practical Apple SDK and supporting new software which requires newer versions of the SDK.

@bergkvist
Copy link
Member

bergkvist commented Jul 18, 2021

MacOS Catalina vs MacOS Big Sur

I set up 2 virtual machines using OSX-KVM. One with Catalina, and one with Big Sur. Then I installed the latest version of nixpkgs on both VMs.

The Python-version from nixpkgs is exactly the same - but the behaviour when using ctypes.util.find_library is different.

MacOS Catalina

  • OS: macOS Catalina 10.15.7 19H15 x86_64
  • Kernel: 19.6.0
tobias@Catalina ~ % nix-shell -p python39 --run "which python"
/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/bin/python

tobias@Catalina ~ % nix-shell -p python39 --run "python -c 'import ctypes.util; print(ctypes.util.find_library(\"c\"))'"
/usr/lib/libc.dylib

MacOS Big Sur

  • OS: macOS 11.4 20F71 x86_64
  • Kernel: 20.5.0
tobias@BigSur ~ % nix-shell -p python39 --run "which python"                                                          
/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/bin/python

tobias@BigSur ~ % nix-shell -p python39 --run "python -c 'import ctypes.util; print(ctypes.util.find_library(\"c\"))'"
None

@bergkvist
Copy link
Member

bergkvist commented Jul 19, 2021

This might be the reason ctypes.util.find_library has stopped working on Big Sur:

New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

Source: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes

Also, see:

@bergkvist
Copy link
Member

bergkvist commented Jul 19, 2021

Digging into the source code in Python ctypes, we see the following code that is responsible for None being returned:

# /nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/ctypes/macholib/dyld.py

# ...

try:
    from _ctypes import _dyld_shared_cache_contains_path
except ImportError:
    def _dyld_shared_cache_contains_path(*args):
        raise NotImplementedError

# ...

def dyld_find(name, executable_path=None, env=None):
    """
    Find a library or framework using dyld semantics
    """
    for path in dyld_image_suffix_search(chain(
                dyld_override_search(name, env),
                dyld_executable_path_search(name, executable_path),
                dyld_default_search(name, env),
            ), env):

        if os.path.isfile(path):
            return path
        try:
            if _dyld_shared_cache_contains_path(path):
                return path
        except NotImplementedError:
            pass
# ...

Notice that os.path.isfile will work on MacOS Catalina, but it won't work on MacOS Big Sur, since the shared libraries are no longer present as files in the file system.

The _dyld_shared_cache_contains_path-function here seems to be an attempted solution for MacOS Big Sur. Notice that in case this function fails to import on MacOS Big Sur, dyld_find will simply return None.

Let's verify that this import actually fails:

tobias@Catalina ~ % nix-shell -p python39 --run "python -c 'from _ctypes import _dyld_shared_cache_contains_path'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_dyld_shared_cache_contains_path' from '_ctypes' (/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so)
tobias@BigSur ~ % nix-shell -p python39 --run "python -c 'from _ctypes import _dyld_shared_cache_contains_path'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '_dyld_shared_cache_contains_path' from '_ctypes' (/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so)

As expected, we get an import error, which explains why ctypes.util.find_library() always returns None on MacOS Big Sur.

Why find_library works on /usr/bin/python3

_dyld_shared_cache_contains_path seems to exist on both the Catalina and Big Sur versions of Python 3.8.2

tobias@Catalina ~ % /usr/bin/python3 -c 'from _ctypes import _dyld_shared_cache_contains_path'
tobias@BigSur ~ % /usr/bin/python3 -c 'from _ctypes import _dyld_shared_cache_contains_path'

@bergkvist
Copy link
Member

bergkvist commented Jul 19, 2021

It seems like the following define-flag needs to be set when compiling python/cpython:

  • HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH

As you can see here, the function _dyld_shared_cache_contains_path won't exist unless HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH has been defined:
https://github.com/python/cpython/blob/2c2055884420f22afb4d2045bbdab7aa1394cb63/Modules/_ctypes/callproc.c#L1995-L1997

Since /usr/bin/python3 (Python 3.8.2) seems to have been compiled with this flag on both Catalina and Big Sur, it probably wouldn't hurt to enable this in general for the versions of Python from nixpkgs.

It seems the pull request to cpython introducing _dyld_shared_cache_contains_path (https://github.com/python/cpython/pull/22855/files) is currently only used in nixpkgs.python39. Probably shouldn't be too hard to also get this into nixpkgs.python38.

@bergkvist
Copy link
Member

bergkvist commented Jul 19, 2021

Python's configure-script seems to automatically define HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH if it can import _dyld_shared_cache_contains_path from <mach-o/dyld.h>.
https://github.com/python/cpython/blob/2c2055884420f22afb4d2045bbdab7aa1394cb63/configure.ac#L3790-L3794

This probably means that if the build server used to populate the nixpkgs cache itself was running on Big Sur, then ctypes.util.find_library would automatically work in nixpkgs.python39.

To better understand why this define-flag exists in the first place, consider the following:

// main.c
#include <stdio.h>
#include <mach-o/dyld.h>

void not_used(void) {
  _dyld_shared_cache_contains_path("test");
}

int main(void) {
  printf("Hello world\n");
  return 0;
}

Trying to compile this on MacOS Big Sur works fine:

tobias@BigSur ctest % clang main.c

But trying to compile it on MacOS Catalina is a different story:

tobias@Catalina ctest % clang main.c
main.c:5:3: error: implicit declaration of function '_dyld_shared_cache_contains_path' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  _dyld_shared_cache_contains_path("test");
  ^
1 error generated.

@bergkvist
Copy link
Member

bergkvist commented Jul 19, 2021

A more polymorphic/cross-compatible alternative to using _dyld_shared_cache_contains_path from mach-o/dyld is to use dlopen, as described in the MacOS Big Sur 11.0.1 release notes:

// library_exists.c
#include <stdio.h>
#include <dlfcn.h>

int library_exists(const char* path) {
    void* handle = dlopen(path, RTLD_LAZY);
    if (handle != NULL) {
        dlclose(handle);
        return 1;
    }
    return 0;
}

int main(int argc, char** argv) {
    const char* path = argv[1];
    if (library_exists(path)) {
        printf("Library (%s) exists\n", path);
    } else {
        printf("Library (%s) not found\n", path);
    }
    return 0;
}

As we can see below, this can be compiled on either Big Sur or Catalina - and does not require any define-flags when compiling

Catalina

tobias@Catalina ctest % clang library_exists.c -o library_exists
tobias@Catalina ctest % ./library_exists /usr/lib/libc.dylib
Library (/usr/lib/libc.dylib) exists
tobias@Catalina ctest % ./library_exists /usr/lib/libc1.dylib
Library (/usr/lib/libc1.dylib) not found

Big Sur

tobias@BigSur ctest % clang library_exists.c -o library_exists
tobias@BigSur ctest % ./library_exists /usr/lib/libc.dylib
Library (/usr/lib/libc.dylib) exists
tobias@BigSur ctest % ./library_exists /usr/lib/libc1.dylib
Library (/usr/lib/libc1.dylib) not found

ctypes change

Imagine that we have wrapped this function into a CPython function importable from _ctypes:

# lib/python3.9/ctypes/macholib/dyld.py
# ...

from _ctypes import library_exists

# ...

def dyld_find(name, executable_path=None, env=None):
    """
    Find a library or framework using dyld semantics
    """
    for path in dyld_image_suffix_search(chain(
                dyld_override_search(name, env),
                dyld_executable_path_search(name, executable_path),
                dyld_default_search(name, env),
            ), env):

        if library_exists(path):
            return path

# ...

@bergkvist
Copy link
Member

Turns out building Python on an older version of MacOS than where you intend on running it is now a legacy/deprecated approach. The way to create a portable binary (since Python 3.9) is to use MACOSX_DEPLOYMENT_TARGET when running configure:

# Support the current version of MacOS that you are compiling on, and all the way back to 10.9
./configure MACOSX_DEPLOYMENT_TARGET=10.9

If my PR to CPython gets merged, it will support the legacy-approach of building on an older MacOS version than the target such that find_library will start working again on Big Sur. Although only for Python >= 3.9.

Python 3.8 has gone into security-fix-only mode, and these changes will as a result never be backported to Python 3.8 on the CPython side. That means Python 3.8 currently is in a kind of limbo-state where you can't make a version that is portable across both Catalina and Big Sur (without patching the source code)

I could create a nixpkgs-specific patch for Python 3.8 to make it portable between Catalina and Big Sur if this is of interest. In particular it would fix the issue of find_library returning None on Big Sur.

@FRidh
Copy link
Member

FRidh commented Aug 6, 2021

Python 3.8 has gone into security-fix-only mode, and these changes will as a result never be backported to Python 3.8 on the CPython side. That means Python 3.8 currently is in a kind of limbo-state where you can't make a version that is portable across both Catalina and Big Sur (without patching the source code)

I could create a nixpkgs-specific patch for Python 3.8 to make it portable between Catalina and Big Sur if this is of interest. In particular it would fix the issue of find_library returning None on Big Sur.

Make the patch, post it on bugs.python.org on the relevant thread, and then fetch it from there.

@bergkvist
Copy link
Member

bergkvist commented Sep 2, 2021

This got backported to Python 3.8 after all: python/cpython#28054

The following releases contain the fix so far:

@larsr
Copy link
Contributor

larsr commented Dec 8, 2021

Fantastic debugging!

Unfortunately for python 3.9 this fix was revoked (for other reasons) in ad1968a

@bergkvist
Copy link
Member

@larsr Although 3.9.6 -> 3.9.7 was reverted for other reasons as you describe, the patch to fix this problem was included for 3.9.6 here: #146477

@larsr
Copy link
Contributor

larsr commented Dec 9, 2021

@bergkvist #146477 was only in the staging branch, not in nixpkgs-unstable or master, and it was removed again six days ago.

The motivation for the removal was that this is fixed in python 3.9.7 (which is true), but history shows that we were unable to just upgrade to 3.9.7.
There must be some communication missing here.
Perhaps the patch should be kept until a successful upgrade to 3.9.7 is possible?

Here is the history for cpython/3.9 in the staging branch
https://github.com/NixOS/nixpkgs/commits/staging/pkgs/development/interpreters/python/cpython/3.9

(There is no patch in nixpkgs-unstable:
https://github.com/NixOS/nixpkgs/tree/nixpkgs-unstable/pkgs/development/interpreters/python/cpython/3.9 )

bergkvist referenced this issue Dec 9, 2021
This patch has been merged in python 3.9.7.

python/cpython@4b55837
@dnadlinger
Copy link
Contributor

So seems like #146477 should be applied again? Or what are the chances for 3.9.9 to be merged soon?

@bergkvist
Copy link
Member

bergkvist commented Dec 22, 2021

Seems like 3.9.9 has already been merged into the staging branch:

python39 = {
sourceVersion = {
major = "3";
minor = "9";
patch = "9";
suffix = "";
};
sha256 = "sha256-BoKMBKVzwHOk5RxCkqJ8G+SuJmIcPtx8+TGEGM47bSc=";
};

Not sure when it will make it into the master branch/unstable release

@dotlambda
Copy link
Member

Can this be closed?
I have an issue in #172274 of which I don't know whether it's related.

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Nov 12, 2022
@abathur
Copy link
Member

abathur commented May 29, 2023

It sounds like this can be closed, and I asked someone in the macOS room on matrix to confirm:

abathur
can anyone on big sur or newer confirm that running the command below yields similar output?

$ nix-shell -p python39 --run "python -c 'import ctypes.util; print(ctypes.util.find_library(\"c\"))'"
/usr/lib/libc.dylib

Zhaofeng Li
Yes, both aarch64-darwin (/nix/store/qb1zy7p74iyzlzj1wz31f7qc9j8ii8vs-python3-3.9.14) and x86_64-darwin (/nix/store/lvswy3rril3p9qzxsa3msfdm35zwwh5g-python3-3.9.16) yielded the same for me
The ARM one is on Ventura 13.5 beta, and X86 one is Ventura 13.4

Happy to reopen if I'm mistaken.

@abathur abathur closed this as completed May 29, 2023
@JohnRTitor JohnRTitor moved this to Big Sur in Darwin Jun 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: darwin Running or building packages on Darwin 6.topic: python
Projects
Status: Big Sur
Development

Successfully merging a pull request may close this issue.