[FR] Make it easier to run cibuildwheel against unlisted Python builds

[webknjaz]

Description

I see that https://github.com/pypa/cibuildwheel/blob/main/cibuildwheel/resources/build-platforms.toml has hardcoded Python versions. In my case I'm building wheels in a separate job with cibuildwheel, store them in GHA artifacts, then download and install in the test job.

And this works well for the stable releases, but not for alphas/nightlies. I'd like an option to run building under arbitrary Python versions not listed there.

One hack I came up with is skipping cibuildwheel and installing the project from source just for py313, in the test job: aio-libs/multidict#909. I don't like it since it doesn't fully replicate what other envs do.

Another hack I saw my colleague do is using a patched version of cibuildwheel (paired with a custom container): https://github.com/python-cffi/cffi/pull/39/files#diff-944291df2c9c06359d37cc8833d182d705c9e8c3108e7cfe132d61a06e9133ddR54-R58 / main...nitzmahone:cibuildwheel:py313_support / https://quay.io/repository/rolpdog/musllinux_1_1_x86_64.

This is because build-platforms.toml is set in stone. One solution could be allowing to specify the configurations on CLI, but that's only usable for building one single wheel at a time, since it's not very friendly to have a lot of CLI args. So a more long-term solution would be allowing to replace or extend that environments declaration.

cc @nitzmahone do you have anything to add?

Build log

No response

CI config

No response

[joerick]

This idea only really works in Linux, where you can set up the manylinux image so that the path you specify in the platform exists in the image. On macOS, the build platform values are used to compute download URLs from Python.org (as well as PKG package identifiers, file system paths). On Windows, everything listed in the build-platforms has to have a published nuget package.

So you'd only be able to add linux configurations. I'm also not sure how big of a win it is...? What is the use of wheels for alphas/nightlies? I think most people advise building everything from source in that scenario, as there are no ABI stability guarantees between versions.

[nitzmahone]

I'm also not sure how big of a win it is

The primary benefit is for projects that do all their testing against built wheels- I've gotten nailed by "all tests pass, but packaged artifact is busted" enough times that I've switched nearly everything I do to that model (meaning cibw is basically the outermost layer of every CI job). When a particular Python/platform combo isn't yet available in cibw, we either need to set up a throwaway build job that manually tries to emulate All The Things cibw does, or use private builds of cibuildwheel + manylinux/musllinux containers that look like the real ones will. I've lately been favoring the latter approach (since then I just change a couple var overrides in my build matrix when cibw releases support for the shiny new thing).

I've had the same wish for Mac/Windows Bring Your Own Python for the same reason- basically "here's an already-installed 3.13 interpreter path- skip the download/setup and just invoke it". I've actually hacked that support in myself in a throwaway fashion a couple times trying to validate e.g., custom Windows ARM64 Python builds, Apple's funky XCode-bundled universal 3.8, with cibuildwheel.

[nitzmahone]

What is the use of wheels for alphas/nightlies?

More than you might think- especially for commonly-used and/or difficult-to-build native-ext deps where absence of wheels can block all but the most determined folks from testing anything that depends on them.

I've tried to hold the line on distributing even pre-releases of such wheels until Python beta/RC, but just having them available as public CI job artifacts unblocks earlier Python testing for a lot of people. Ever tried to build libffi on Windows/ARM64? I don't recommend it... 😆

[joerick]

Interesting. I'm wondering if that there is an approach here that only needs a path to an installed interpreter. We currently have PythonConfiguration objects in the codebase, it should be possible to rename these BuiltinPythonConfiguration and have some UserPythonConfiguration which would work similarly but skip the install steps.

So an interface to this could work something like

--add-python ID:PATH
e.g. 
cibuildwheel --add-python my-cp313-macosx_arm64:~/interpreters/cpython313/bin/python3 --only my-cp313-macosx_arm64

[nitzmahone]

Yeah, that's exactly where I hacked it in for Mac/Windows when I was just screwing around and needed it fast... I never submitted a PR because it felt like the config was missing a necessary abstraction to do this cleanly, which your proposal neatly solves. Should meet my needs for the Mac/Windows side, and could probably also accommodate the Linux container side without too much trouble as well, since the container image is already overridable, so as long as it can still find an arbitrary Python by path in the container, I can just continue using my private builds of many/musllinux containers with the super-pre-release Pythons grafted in.

One random thought on that: since it's not exposing the download/install, will/should CIBW_BEFORE_BUILD function if the listed interpreter isn't yet installed (e.g., so I could use that hook to install python.org Mac/Windows pre-releases)? When I hacked it in for 3.13 this time, I actually just relied on the existing published pre-release builds for those, but if I had to choose, I'd much prefer completely arbitrary "BYO Python by path" to fully dynamic PythonConfiguration entries that require an installer. IIRC, today that'd be a chicken/egg problem, since the listed interpreter path would be missing and there'd be no way to install it, so that build would never be allowed to run. No big deal if that's too difficult to work around- I can still just add a conditional matrix task to handle the install ahead of time if the BYO path is supported.

Also happy to take a whack at building some form of that, since it sounds like you're amenable to the idea...

Thanks to you and the rest of the cibw team for all your efforts, BTW- it's been so nice to increasingly lean on cibw without needing to reinvent so many wheels (no pun intended).

[webknjaz]

So you'd only be able to add linux configurations.

That's fine for starters. In the workflows with the following CI structure

build sdist -> build a matrix of wheels -> test those wheels without Git checkout even -> publish

the ability to run the experimental jobs the same way as the normal ones is a deal breaker. It'll reduce the maintenance burden if things can be unified.

I'm also not sure how big of a win it is...?

Later on, maybe it'll be possible to provision Python differently even on macOS / Windows. Or just fall back to what GH ships, keeping in mind that those wheels would be just for testing and shouldn't be published to the regular PyPI.

This can also help people get started with #1657 once it's supported.

What is the use of wheels for alphas/nightlies?

The win is to let other projects run some testing ahead of time without having to worry about setting up a suitable build env. Here's an example of using dumb-pypi in one of the projects I got: https://github.com/ansible/pylibssh#nightlies--dumb-pypi--github-pages

[webknjaz]

since the container image is already overridable, so as long as it can still find an arbitrary Python by path in the container,

Does this approach allow for not yet existing wheel tags? I there's going to be a t suffix for nogil, for example. Is there any validation that would prevent it from accepting arbitrary identifiers?

[mayeut]

Is there any validation that would prevent it from accepting arbitrary identifiers?

They do have a semantic so, while they're not checked against a known list, they have to respect some semantic or different exceptions will be raised and/or the workflow won't be valid.

You can add a missing versions easily (e.g. cp314-manylinux_x86_64), that would work.
I'd say accepted identifier look like [cp]p\d\d+-(platform_list)

The t suffix for nogil required some changes in the code in order for cibuildwheel to work. c.f. #1831. Accepted identifiers becomes [cp]p\d\d+t?-(platform_list).

• start with 2 letters, either "cp" or "pp", enforced by the logger (& used in the workflow):
  

  cibuildwheel/cibuildwheel/logger.py

  Lines 206 to 212 in 78bca57

  	if python_interpreter == "cp":
  	build_description += "CPython"
  	elif python_interpreter == "pp":
  	build_description += "PyPy"
  	else:
  	msg = f"unknown python {python_interpreter!r}"
  	raise Exception(msg)

• followed by the non-dotted major/minor version:
  

  cibuildwheel/cibuildwheel/util.py

  Lines 246 to 249 in 78bca57

  	if self.requires_python is not None:
  	py_ver_str = build_id.split("-")[0]
  	major = int(py_ver_str[2])
  	minor = int(py_ver_str[3:])

  
  

  cibuildwheel/cibuildwheel/util.py

  Lines 660 to 662 in 78bca57

  	if tag.interpreter != "py3" and int(tag.interpreter[3:]) > int(interpreter[3:]):
  	# If a minor version number is given, it has to be lower than the current one.
  	continue

• followed by -{platform}. The platform must be in a known list, enforced by the logger. It's used in the workflow to install the correct python (Windows/macOS), start the correct docker image (Linux), know what's cross-compilation/can be tested (Windows/macOS):
  

  cibuildwheel/cibuildwheel/logger.py

  Lines 216 to 220 in 78bca57

  	try:
  	build_description += PLATFORM_IDENTIFIER_DESCRIPTIONS[platform_identifier]
  	except KeyError as e:
  	msg = f"unknown platform {platform_identifier!r}"
  	raise Exception(msg) from e

  
  

  cibuildwheel/cibuildwheel/logger.py

  Lines 20 to 37 in 78bca57

  	PLATFORM_IDENTIFIER_DESCRIPTIONS: Final[dict[str, str]] = {
  	"manylinux_x86_64": "manylinux x86_64",
  	"manylinux_i686": "manylinux i686",
  	"manylinux_aarch64": "manylinux aarch64",
  	"manylinux_ppc64le": "manylinux ppc64le",
  	"manylinux_s390x": "manylinux s390x",
  	"musllinux_x86_64": "musllinux x86_64",
  	"musllinux_i686": "musllinux i686",
  	"musllinux_aarch64": "musllinux aarch64",
  	"musllinux_ppc64le": "musllinux ppc64le",
  	"musllinux_s390x": "manylinux s390x",
  	"win32": "Windows 32bit",
  	"win_amd64": "Windows 64bit",
  	"win_arm64": "Windows on ARM 64bit",
  	"macosx_x86_64": "macOS x86_64",
  	"macosx_universal2": "macOS Universal 2 - x86_64 and arm64",
  	"macosx_arm64": "macOS arm64 - Apple Silicon",
  	}