Eliminate the use of deprecated OS X APIs in getpath.c

[ned-deily]

BPO	15498
Nosy	@ronaldoussoren, @vstinner, @ned-deily
Files	issue15498-v1.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = None
created_at = <Date 2012-07-30.07:34:24.426>
labels = ['OS-mac', '3.11']
title = 'Eliminate the use of deprecated OS X APIs in getpath.c'
updated_at = <Date 2021-11-27.11:57:12.234>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2021-11-27.11:57:12.234>
actor = 'iritkatriel'
assignee = 'ronaldoussoren'
closed = False
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2012-07-30.07:34:24.426>
creator = 'ned.deily'
dependencies = []
files = ['26622']
hgrepos = []
issue_num = 15498
keywords = ['needs review']
message_count = 4.0
messages = ['166863', '166991', '173828', '354378']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue15498'
versions = ['Python 3.11']

Linked PRs

• gh-59703: use the dladdr instead of deprecated macOS APIs in getpath.c #111546
• gh-59703: restore include of mach-o/dyld.h  #112309

[ned-deily]

getpath.c uses three OS X APIs that have been producing deprecation warnings since at least OS X 10.5: NSModuleForSymbol, NSLookupAndBindSymbol, and NSLibraryNameForModule. We should figure out how to live without them.

[ronaldoussoren]

Removing the dependency on NSLookupAndBindSymbol (and related APIs) is easier than I expected.

The attached patch (bpo-15498-v1.txt) uses dladdr to get symbol information for Py_Initialize and that information includes the path for the library where that symbol is located (that is the framework).

The patch has seen only light testing, in particular: I've verified that the library path has the expected value, but haven't run the full test suite set.

TODO:

• Run test full test suite with a framework build (both before and after
  installing)
• Likewise for a regular unix build
• Likewise for a regular unix build with --enable-shared

This patch should work on OSX 10.4 or later, and may also work on OSX 10.3, the manpage for dladdr is not entirely clear on whether or not the symbol is available in libSystem on OSX 10.3.

[ronaldoussoren]

Apple seems to have switched to using dlopen in their version of getpath.c (see <http://www.opensource.apple.com/source/python/python-60.3/2.7/fix/getpath.c.ed\>, this is the version in OSX 10.8.2)

This causes a problem for some people, as noticed on the pythonmac mailing list. This is something we should test before applying my patch to avoid regressions.

Message-Id: <[email protected]> says:

I am trying to work with Apple Mountain Lion's install of Python 2.7. I
have a language interoperability tool, Babel
http://www.llnl.gov/CASC/components/, that used embedded Python when
other languages are calling Python (c.g., C++ calling Python). My
fundamental problem is that sys.path is not being initialized the same
when I dlopen(libpython2.7.dylib) and initialize Python compared with
how the sys.path is set when the Python executable is called directly.
This causes Python to fail to load the numpy module used by Babel.

bash-3.2$ /usr/bin/python2.7 -c "import sys; print sys.path; import
numpy"> /tmp/out1
bash-3.2$ /usr/bin/python -c "import sys; print sys.path; import numpy"
/tmp/out2
bash-3.2$ cat /tmp/out1
['',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
'/Library/Python/2.7/site-packages']
bash-3.2$ diff /tmp/out1 /tmp/out2
bash-3.2$ ls -al /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Aug 23 11:10 /usr/bin/python2.7 ->
../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

Presumably, this C program that uses dlopen(), Py_Initialize, and
Py_SimpleString should have exactly the same output.

/** foo.c */
#include<dlfcn.h>
#include<stdio.h>

int
main(int argc, char **argv)
{
 // void *lptr =
dlopen("/System/Library/Frameworks/Python.framework/Python", RTLD_NOW |
RTLD_GLOBAL);
 void *lptr =
dlopen("/System/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib",
RTLD_NOW | RTLD_GLOBAL);
 if (lptr) {
   void (*pyinit)(void) = dlsym(lptr, "Py_Initialize");
   if (pyinit) {
     int (*runSimple)(const char *);
     (*pyinit)();              /* initialize Python */
     runSimple = dlsym(lptr, "PyRun_SimpleString");
     if (runSimple) {
       (*runSimple)("import sys ; print sys.path; import numpy");
     }
   }
   else {
     fprintf(stderr, "Unable to locate Py_Initialize: %s\n", dlerror());
   }
 }
 else {
   fprintf(stderr, "Error loading Python shared library: %s\n",
dlerror());
 }
}

bash-3.2$ gcc foo.c ; ./a.out
['/usr/lib/python27.zip', '/usr/lib/python2.7',
'/usr/lib/python2.7/plat-darwin', '/usr/lib/python2.7/plat-mac',
'/usr/lib/python2.7/plat-mac/lib-scriptpackages',
'/usr/Extras/lib/python', '/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload']
Traceback (most recent call last):
 File "<string>", line 1, in<module>
ImportError: No module named numpy

However as you see, it has a completely different sys.path. I can't seem
to find a way to get it to initialize Python with the same sys.path. It
seems like the libpython2.7.dylib is broken. I am at a loss on how to
fix this or even who to ask for help.

[vstinner]

The deprecation warnings are not fixed yet. They were quickly mentioned at:
https://bugs.python.org/issue38429#msg354367

[ned-deily]

Thanks for the PR, @aisk! Merged for release in 3.13.0a2.

[ned-deily]

Apparently I didn't test the PR enough. It turns out that the include of mach-o/dyld.h is needed to build on older versions of macOS, at least on macOS 10.13 and earlier. Otherwise, getpath.c fails with something similar to:

./Modules/getpath.c:730:13: error: implicit declaration of function '_NSGetExecutablePath' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]

I'm not quite sure why it compiles without error on more recent versions of macOS but, looking at the Apple Open Source site it does appear that they made some significant updates to dyld in more recent releases.

[chrstphrchvz]

For a long time, the synopsis section of man 3 dyld has said to include <mach-o/dyld.h> for the function declaration of _NSGetExecutablePath(). This example program does not include that header:

#include <stdint.h>
int main(void){
    char buf[1];
    uint32_t bufsize = 0;
    return _NSGetExecutablePath(buf, &bufsize);
}

Compiling this program with LLVM.org Clang 17.0.6 (obtained from MacPorts) emits the expected implicit function declaration error. But compiling it with recent AppleClang (e.g. from Xcode 14.2 or Command Line Tools 15.0) succeeds without warnings or errors. I do not know if this expected AppleClang behavior for _NSGetExecutablePath(), or an AppleClang bug. But it is arguably a portability issue and inconsistent with documentation.