get_supported() should return more possibilities

[agronholm]

Originally reported by: Antoine Pitrou (Bitbucket: pitrou, GitHub: pitrou)

---

The following result isn't compatible with a tag such as py2.py3-none-linux_x86_64.
Reference: https://mail.python.org/pipermail/distutils-sig/2014-October/025176.html

Python 3.4.1 (3.4:d1bf37def4fd, May 19 2014, 19:52:53) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wheel
>>> from wheel.pep425tags import get_supported
>>> get_supported()
[('cp34', 'cp34m', 'linux_x86_64'), ('cp34', 'abi3', 'linux_x86_64'), ('cp34', 'none', 'linux_x86_64'), ('cp34', 'none', 'any'), ('cp3', 'none', 'any'), ('cp33', 'none', 'any'), ('cp32', 'none', 'any'), ('cp31', 'none', 'any'), ('cp30', 'none', 'any'), ('py34', 'none', 'any'), ('py3', 'none', 'any'), ('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none', 'any'), ('py30', 'none', 'any')]

---

• Bitbucket: https://bitbucket.org/pypa/wheel/issue/125

[agronholm]

What kind of project would produce such wheels anyway?

[agronholm]

To answer my own question: milksnake produces such wheels.

[ncoghlan]

Technically any package could use this capability - all they need to do is to bundle a platform-dependent binary, which they then interact with via Python-independent interfaces (e.g. launch via subprocess, and then communicate via pipes and/or sockets).

But yeah, the more likely case for it coming up is when an extension module depends on a compatibility shim that handles the interaction with the CPython runtime, so only that shim needs to be rebuilt when the CPython ABI changes.

[agronholm]

Ok, so making sure that pyX-none-$platform is included should suffice, yes?

[ncoghlan]

Yeah, I think that's the key missing combination - platform dependent, but not Python ABI dependent.

[agronholm]

Ok, what about the limited ABI? Should such wheels be named like -py3-abi3-$platform.whl? I didn't see anything in the spec about the limited ABI requiring CPython specifically, although it doesn't look like there's any other implementation that supports it.

[ncoghlan]

Aye, I think it would make sense to allow that combination as well, as it would allow the publisher to indicate that their Python code was implementation independent, and it was only their extension modules that required a particular runtime ABI.

[agronholm]

Wheel already includes pyX-none-$platform in the list of supported tags and has done so since November of 2014: https://github.com/pypa/wheel/blob/master/wheel/pep425tags.py#L176
Now we just need a way to determine that it contains platform specific binaries. How about scanning through the list of included files and turning on the binary flag if any dynamic libraries are included (.so on linux, .dll on Windows and .dylib on macOS)? The same method could be used to automatically detect ABI3 wheels, removing the need for the --py-limited-abi switch.

[ncoghlan]

I don't have a strong opinion on the automated scanning - in any case where I've needed this, it's been as part of a build script that could pass the required settings to force wheel to include the platform tag (i.e. the #153 case).

[dholth]

I have always thought that automated scanning of file contents was far, far too clever. Scanning of setuptools Extension objects is somewhere between just clever enough or too clever. Did we ever grow a flag that means 'the last tag should be whichever platform we are compiling on now'?

[dholth]

For example bdist_wheel could know that abi3 dlls on Windows have the same filenames as non-abi3 dlls, but link with PYTHON3.DLL (which we could determine by running the Windows equivalent of objdump or ldd on the file) but on Linux abi3 dlls probably have a different extension. OSX probably does something else. Can we dump all the symbols from the library to determine whether they are using abi3 correctly or accidentally referenced a non-abi3 api?
Or, assuming the user knows they are trying to publish an abi3 extension, they could pass a flag.

[agronholm]

I suggested autodetection because I was ignorant about how ABI3 works. While there is an easy way to detect if ABI3 was used, the proper Python tags are very hard to determine automatically, which is I guess why the --py-limited-abi option was added in the first place. I don't think we have any reliable way to determine the Python tag(s) automatically.

[agronholm]

Now that we've delegated this functionality to packaging, I think I can close this issue.