-
-
Notifications
You must be signed in to change notification settings - Fork 14.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python's ctypes broken on macOS Big Sur #105038
Comments
That's a pretty big backport! Probably in time it will be backported to 3.8 as well. |
once they do another release, these will be added "automatically" |
Has this actually been resolved in any new release?
3.8.7:
And 3.9:
|
Getting a similar error when starting
My guess is that it has something to do with the patch to ctypes being applied to all the python derivations. |
Upsteam hasn't cut any new releases yet. |
But cpython 3.9.1 (which is in nixpkgs-unstable) should work on Big Sur since it has the patch from python/cpython#23295, right? |
For reference, the latest Python 3.9 from Homebrew works on Big Sur
|
Nevermind, I was distracted while reading through the Homebrew's commit history, this has been removed in their 3.9.1 update. I'm still not sure why Homebrew's Python 3.9.1 works but not nix |
So Python 3.9.1 should have ctypes work on Big Sur. Nix's Python still doesn't work because nixpkg's build doesn't provide python/cpython@4176193#diff-90d08e583c4c9c6f391b2ae90f819f600a6326928ea9512c9e0c6d98e9f29ac2R12040 I'm checking if there is any way to fix it. (I saw there is |
Oh well, this is a fun digging. So apparently dyld in nixpkgs was last updated in macOS 10.12 release to 433.5. The function used in fixing Python's ctypes issue appeared in dyld-832.7.1.tar.gz. Update: now I realized simply updating dyld will not work since it depends on xnu (because of the My messy overlay file and patches (note this is very reckless so probably only use if you know what it means): https://gist.github.com/fanzeyi/3af0396eb5cfaa81ad30af427b1e8069 |
@FRidh As a maintainer of the cpython package, do you know if there are steps towards a resolution or issues we should be tracking here? I do of course realise that cpython on Darwin is a challenge. If we have some plans or comms, then perhaps the community can assist in some way. |
@groodt I don't work with darwin and I am not aware of any PR's either towards adding support for big sur. |
@LnL7 @jonringer @domenkozar Do we have anywhere public where issues such as this are prioritised? It feels like this Python issue is only going to be the start of other major blockers for darwin as more and more packages expect newer versions of Apple SDKs. I fully understand that macOS support for Nix is always a struggle, just perhaps hoping that there is an issue being tracked around Big Sur support. Not to mention M1 Apple Silicon support at some stage. |
@groodt As part of https://opencollective.com/nix-macos the goal is also to come up with a central place to track macOS support in a systematic way. |
@groodt, #105026 tracks Apple Silicon support. I will be working towards tracking the status of macOS support. There is a great deal of friction between supporting the broadest possible userbase, which requires a stdenv based on the oldest practical Apple SDK and supporting new software which requires newer versions of the SDK. |
MacOS Catalina vs MacOS Big SurI set up 2 virtual machines using OSX-KVM. One with Catalina, and one with Big Sur. Then I installed the latest version of nixpkgs on both VMs. The Python-version from nixpkgs is exactly the same - but the behaviour when using ctypes.util.find_library is different. MacOS Catalina
tobias@Catalina ~ % nix-shell -p python39 --run "which python"
/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/bin/python
tobias@Catalina ~ % nix-shell -p python39 --run "python -c 'import ctypes.util; print(ctypes.util.find_library(\"c\"))'"
/usr/lib/libc.dylib MacOS Big Sur
tobias@BigSur ~ % nix-shell -p python39 --run "which python"
/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/bin/python
tobias@BigSur ~ % nix-shell -p python39 --run "python -c 'import ctypes.util; print(ctypes.util.find_library(\"c\"))'"
None |
This might be the reason ctypes.util.find_library has stopped working on Big Sur:
Source: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes Also, see: |
Digging into the source code in Python ctypes, we see the following code that is responsible for # /nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/ctypes/macholib/dyld.py
# ...
try:
from _ctypes import _dyld_shared_cache_contains_path
except ImportError:
def _dyld_shared_cache_contains_path(*args):
raise NotImplementedError
# ...
def dyld_find(name, executable_path=None, env=None):
"""
Find a library or framework using dyld semantics
"""
for path in dyld_image_suffix_search(chain(
dyld_override_search(name, env),
dyld_executable_path_search(name, executable_path),
dyld_default_search(name, env),
), env):
if os.path.isfile(path):
return path
try:
if _dyld_shared_cache_contains_path(path):
return path
except NotImplementedError:
pass
# ... Notice that The Let's verify that this import actually fails: tobias@Catalina ~ % nix-shell -p python39 --run "python -c 'from _ctypes import _dyld_shared_cache_contains_path'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: cannot import name '_dyld_shared_cache_contains_path' from '_ctypes' (/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so) tobias@BigSur ~ % nix-shell -p python39 --run "python -c 'from _ctypes import _dyld_shared_cache_contains_path'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: cannot import name '_dyld_shared_cache_contains_path' from '_ctypes' (/nix/store/1z0mfkg7hvdh0yzv0gljxpqplv69b4mk-python3-3.9.4/lib/python3.9/lib-dynload/_ctypes.cpython-39-darwin.so) As expected, we get an import error, which explains why Why find_library works on /usr/bin/python3_dyld_shared_cache_contains_path seems to exist on both the Catalina and Big Sur versions of Python 3.8.2 tobias@Catalina ~ % /usr/bin/python3 -c 'from _ctypes import _dyld_shared_cache_contains_path' tobias@BigSur ~ % /usr/bin/python3 -c 'from _ctypes import _dyld_shared_cache_contains_path' |
It seems like the following define-flag needs to be set when compiling python/cpython:
As you can see here, the function Since /usr/bin/python3 (Python 3.8.2) seems to have been compiled with this flag on both Catalina and Big Sur, it probably wouldn't hurt to enable this in general for the versions of Python from nixpkgs. It seems the pull request to cpython introducing |
Python's configure-script seems to automatically define This probably means that if the build server used to populate the nixpkgs cache itself was running on Big Sur, then ctypes.util.find_library would automatically work in nixpkgs.python39. To better understand why this define-flag exists in the first place, consider the following: // main.c
#include <stdio.h>
#include <mach-o/dyld.h>
void not_used(void) {
_dyld_shared_cache_contains_path("test");
}
int main(void) {
printf("Hello world\n");
return 0;
} Trying to compile this on MacOS Big Sur works fine: tobias@BigSur ctest % clang main.c But trying to compile it on MacOS Catalina is a different story: tobias@Catalina ctest % clang main.c
main.c:5:3: error: implicit declaration of function '_dyld_shared_cache_contains_path' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
_dyld_shared_cache_contains_path("test");
^
1 error generated. |
A more polymorphic/cross-compatible alternative to using // library_exists.c
#include <stdio.h>
#include <dlfcn.h>
int library_exists(const char* path) {
void* handle = dlopen(path, RTLD_LAZY);
if (handle != NULL) {
dlclose(handle);
return 1;
}
return 0;
}
int main(int argc, char** argv) {
const char* path = argv[1];
if (library_exists(path)) {
printf("Library (%s) exists\n", path);
} else {
printf("Library (%s) not found\n", path);
}
return 0;
} As we can see below, this can be compiled on either Big Sur or Catalina - and does not require any define-flags when compiling Catalinatobias@Catalina ctest % clang library_exists.c -o library_exists
tobias@Catalina ctest % ./library_exists /usr/lib/libc.dylib
Library (/usr/lib/libc.dylib) exists
tobias@Catalina ctest % ./library_exists /usr/lib/libc1.dylib
Library (/usr/lib/libc1.dylib) not found Big Surtobias@BigSur ctest % clang library_exists.c -o library_exists
tobias@BigSur ctest % ./library_exists /usr/lib/libc.dylib
Library (/usr/lib/libc.dylib) exists
tobias@BigSur ctest % ./library_exists /usr/lib/libc1.dylib
Library (/usr/lib/libc1.dylib) not found ctypes changeImagine that we have wrapped this function into a CPython function importable from # lib/python3.9/ctypes/macholib/dyld.py
# ...
from _ctypes import library_exists
# ...
def dyld_find(name, executable_path=None, env=None):
"""
Find a library or framework using dyld semantics
"""
for path in dyld_image_suffix_search(chain(
dyld_override_search(name, env),
dyld_executable_path_search(name, executable_path),
dyld_default_search(name, env),
), env):
if library_exists(path):
return path
# ... |
Turns out building Python on an older version of MacOS than where you intend on running it is now a legacy/deprecated approach. The way to create a portable binary (since Python 3.9) is to use MACOSX_DEPLOYMENT_TARGET when running configure: # Support the current version of MacOS that you are compiling on, and all the way back to 10.9
./configure MACOSX_DEPLOYMENT_TARGET=10.9 If my PR to CPython gets merged, it will support the legacy-approach of building on an older MacOS version than the target such that find_library will start working again on Big Sur. Although only for Python >= 3.9. Python 3.8 has gone into security-fix-only mode, and these changes will as a result never be backported to Python 3.8 on the CPython side. That means Python 3.8 currently is in a kind of limbo-state where you can't make a version that is portable across both Catalina and Big Sur (without patching the source code) I could create a nixpkgs-specific patch for Python 3.8 to make it portable between Catalina and Big Sur if this is of interest. In particular it would fix the issue of find_library returning None on Big Sur. |
Make the patch, post it on |
This got backported to Python 3.8 after all: python/cpython#28054 The following releases contain the fix so far:
|
Fantastic debugging! Unfortunately for python 3.9 this fix was revoked (for other reasons) in ad1968a |
@bergkvist #146477 was only in the staging branch, not in nixpkgs-unstable or master, and it was removed again six days ago. The motivation for the removal was that this is fixed in python 3.9.7 (which is true), but history shows that we were unable to just upgrade to 3.9.7. Here is the history for cpython/3.9 in the staging branch (There is no patch in nixpkgs-unstable: |
This patch has been merged in python 3.9.7. python/cpython@4b55837
So seems like #146477 should be applied again? Or what are the chances for 3.9.9 to be merged soon? |
Seems like 3.9.9 has already been merged into the staging branch: nixpkgs/pkgs/development/interpreters/python/default.nix Lines 132 to 140 in 373d234
Not sure when it will make it into the master branch/unstable release |
Can this be closed? |
It sounds like this can be closed, and I asked someone in the macOS room on matrix to confirm:
Happy to reopen if I'm mistaken. |
Python's ctypes support is broken on macOS Big Sur. This currently affects nixpkgs. Here's an example with
python3.8-regex
:https://gist.github.com/dhess/e1b2662a6732618b252096e0de3a04d4
This has been fixed in CPython trunk and backported to 3.9:
python/cpython#22855
python/cpython#23295
Unfortunately, it's not yet been backported to 3.8, which is the default
python3Packages
set innixpkgs
.The text was updated successfully, but these errors were encountered: