We should probably allow libz linkage, and maybe libexpat

[njsmith]

Here's the output of ldd python3 on my system (Ubuntu 18.10):

~$ ldd /usr/bin/python3
	linux-vdso.so.1 (0x00007ffd9ddbb000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1f0e73b000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1f0e71a000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1f0e714000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f1f0e70f000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f1f0e6d2000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1f0e4b5000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1f0e326000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1f0e94c000)

Most of that is glibc or kernel stuff. The exceptions are libexpat.so.1, and libz.so.1.

ldd /usr/bin/python2 gives a similar list, except it's missing libexpat.so.1.

The standard auditwheel profiles don't allow wheels to link against libz.so.1 or libexpat.so.1 – they get vendored instead. (See also: pypa/manylinux#296)

However... this doesn't actually work!

Because of how ELF symbol resolution works, it doesn't matter whether you vendor these libraries: if they show up in ldd <the interpreter>, then any symbols in the system version of those libraries will be used by all python extensions, no matter what they vendor or link against. Basically the system version of libz.so.1 and libexpat.so.1 are getting LD_PRELOADed into every extension module.

And apparently... this works? I haven't seen any bugs about it. And it's plausible to me that these libraries have extremely stable ABIs.

So maybe we should add them to the list of libraries that auditwheel will allow you to depend on without vendoring... the vendoring isn't doing much anyway.

[mayeut]

The dump for python interpreter on manylinux1 and manylinux2010 images are not quit the same.
They only rely on .so mentioned in the respective PEP except for python 3.7 on manylinux2010 which also adds:

libfreebl3.so => /lib64/libfreebl3.so (0x00007fd024df0000)

That being said, I do agree that at least libz.so.1 should be added to the list. zlib is a package that's installed on almost all linux distros as a default package (if not all, it's always available and is a python dependency anyway, at least for the zlib module if not for the interpreter). Its ABI is very stable indeed.

About libexpat.so.1, well, it's not available through yum on manylinux1 (centos5) but is available on manylinux2010 (centos6). I don't know much about its ABI stability.

[root@ce48f06f97b0 /]# yum provides '*/libexpat.so.1'
Reducing CentOS-5 - libselinux to included packages only
Finished
No Matches found

If its ABI is stable, then vendoring it might still be the way to go, it would be unused on systems having a python interpreter linking against it but still be used on systems without it.

[jjhelmus]

expat has been ABI compatible since the 2.0 release where there was a SONAME change. CentOS 5 includes expat 1.95 where as CentOS 6 has 2.0.1.

[matthew-brett]

Are there any Pythons that do not link against libz and libexpat?

How should we fix this properly? I guess we have to confirm ABI compatibility for all versions of libz and libexpat that may be linked to any installed Python.

[jjhelmus]

Are there any Pythons that do not link against libz and libexpat?
There are Pythons that do not link against libexpat, for example the ones provided by Anaconda, Inc and conda-forge. The pyexpat module can be build against the system expat library or not using the--with-system-expat argument. Both Anaconda, Inc and conda-forge do not use this argument Linking against libz should be more reliable although it is possible to statically link libz.

[mattip]

Can we separate this into two: remove libz and continue exploring libexpat?

[mayeut]

@mattip, in fact, having checked both, removing libexpat seems safer than libz for now. While I don't expect anything going wrong with libz fo most users, its ABI is in fact less stable than libexpat, with symbols added/removed at different points in time for the same soname (while it makes sense for added symbols, it doesn't for removed ones...).

[asottile-sentry]

hmm so slightly annoyingly the official python slim docker images don't contain libexpat.so.1:

$ docker run --rm -ti python:3.12.6-slim bash -c 'ls /lib/*/libexpat'
ls: cannot access '/lib/*/libexpat': No such file or directory