Setuptools build shim doesn't actually install script

[alerque]

I'm trying to upgrade the Arch Linux AUR package to 53.0 and something isn't right. Because it's still basically the only Python ecosystem that manages installs to alternate roots correctly almost all package builds use setuptools. The package as downloaded from PyPi does have a flit generated setup.py file. To my eye it even looks right, it has the packages and entry_points that I would expect to see. However when running python setup.py install --root=... I do not get any /usr/bin/weasyprint script as everything up through 52.x has been doing and I would expect to see now.

[liZe]

Hello!

The problem is described in pypa/flit#380, pypa/packaging#363, and Python’s Discuss.

Spoiler alert: ⚠ People are often angry about this topic ⚠.

TL;DR: installing a Python package without building a wheel is not standardized, and distributions are on their own to support PEP 517 😒. The Python packaging authority spent a lot of time to create standards, some developers spent a lot of time to follow these standards in their new packaging tools, and distribution packagers complain because these new standards are different from the crazy hacks they had to write to make Python packages installable when there was no standard.

We’re really sorry about this situation. Unfortunately, it looks like there’s no obvious solution right now to reliably install packages without building a wheel. Gentoo built a tool called pyproject2setuppy, trying to transform pyproject.toml files into setup.py files that work with the way Gentoo installs packages.

The "real" solution proposed to distributions by the Python Packaging Authority is:

• use the packaged wheel when available, or build a wheel from the source using standard tools,
• install this wheel (or create a distribution package) with a tool specifically written to fulfill the distribution’s needs.

I know that it’s hard for packagers to follow this advice, because they already wrote a lot of code to make setuptools follow their needs. Rewriting a wheel installer from scratch is sad 😢. But wheels being standardized makes this solution more reliable than the previous setuptools-based hacks. It’s a lot of work, but it’s probably the way to go…

[liZe]

• install this wheel (or create a distribution package) with a tool specifically written to fulfill the distribution’s needs.

installer seems to be a really nice way to achieve this step.

[alerque]

Wow, just wow. So much missing the point! Nobody is down on the tooling rewrites, we just need a tool. To quote something buried in those threads:

It's downright depressing to see this hostile attitude toward distros that are literally asking for one thing and one thing only: do not drop support for the old way of doing things until after a new way is implemented.

That was two years ago and the issues is still not addressed. The new tooling handles the user-space and venv style installations nicely, but does not meet this use case. The installer and other work-arounds have huge drawbacks and are clearly written as if the thing they are doing shouldn't be done.

For anybody else hitting this issue for this package, this problem can be fixed by monkey patching the Flit generated script. It is loading the even more legacy distutils as a shim but passing it data meant for setuputils! It works fine and the binary is installed if you just swap out the library for the one that Flit should be importing all along. The upstream rational for leaving this clear mistake is just weird (and probably more political than technical but I don't feel like going deeper down the rabbit hole).

sed -i -e 's/distutils.core/setuptools/' setup.py

[liZe]

It's downright depressing to see this hostile attitude toward distros that are literally asking for one thing and one thing only: do not drop support for the old way of doing things until after a new way is implemented.

I agree, that’s really sad. The PyPA says that it’s the distribution’s job to create these tools (because each distribution has its own needs), the distributions packagers say that it’s the PyPA’s job to provide everything to install wheels without pip. Everybody’s waiting and complaining 😢.

For anybody else hitting this issue for this package, this problem can be fixed by monkey patching the Flit generated script. It is loading the even more legacy distutils as a shim but passing it data meant for setuputils! It works fine and the binary is installed if you just swap out the library for the one that Flit should be importing all along. The upstream rational for leaving this clear mistake is just weird (and probably more political than technical but I don't feel like going deeper down the rabbit hole).

It looks like a clear mistake from the distribution’s point of view, I agree. But it looks like the way to go for packagers, both from a technical and a political point of view. I’ll try to explain this in this comment, so that each side can understand a bit more the other one (I know, that’s a dream 😉).

This setup.py file in the source file is really here for legacy tools. pip doesn’t need it. wheel doesn’t need it.

From the Python team’s point of view, the installation of packages in distributions should follow these steps:

• (optional, it’s often already done by the project maintainer) build a wheel from the source package,
• install the wheel on the system.

For the Python developers and for the projects maintainers, there’s no need for a setup.py for these steps when the package follows standard steps for building and installation. pyproject.toml is enough.

But the question of the distribution packagers remains: why does the Python team changes something that works (i.e. setuptools and setup.py)?

Again, I think that this question is legitimate. Setuptools has done its job quite well. Moreover, and they may have forgotten this, but packagers spent a lot of time and energy to make setuptools work for the job. Many tools have been written, many hacks have been included in packages, an insane number of bugs have been reported and fixed in distutils, setuptools, pip… It’s true to say that it now somehow works.

But… Let’s be fair. Before pyproject.toml was introduced, everybody complained about the mess Python packaging was. There was no clear specification, and setuptools is full of bugs, full of legacy code, full of commands, full of options, full of formats… PEP 518 explains the reasons why better than I would, and we have to admit that a standard is often better than the "code is law" philosophy we used to have.

From my point of view, as a project maintainer, that’s a huge step in the right direction. Packaging and releasing is a pleasure with Flit (the same applies to pipenv lovers). Having all the options for all the tools we use in just one file is great. I don’t have to play with a random setuptools undocumented option, hoping that this "feature" (or is it a bug, nobody knows when code is law) will be supported in the future.

From my point of view, as an amateur packager (yes, I have both hats 😝), it’s really frightening. I have a long experience in setuptools, I have written a lot of hacks and used a lot of useful options that I don’t want to lose. And there’s no out-of-the-box solution to install wheels.

So, it means that I have to write my own tools to install wheels the way my distribution wants. Again. That’s not fair 😒.

Everybody will have to find the courage to do this together. I think that PyPA has to provide a library helping distributions, because they decided to introduce all these changes (because almost everybody asked for them, of course 😄). Installer, written by a PyPA member, may be the "answer". But, as it’s written by the PyPA, it will need a lot of help from packagers to understand and what they want (and they want different things depending on the distribution, of course). It’s far from perfect, it should have been written before, that’s true. But we really have to help them turn this seed into a success.

I deeply think that installing from wheels is a great idea, because everything is already prepared for installation, and the format is specified. I know it’s hard to admit. It’s hard to admin for me, at least the "distribution packager" part. It requires some time and a lot of constructive talks between the PyPA and the distributions. But when it’s done, packagers will be able to forget their hacks and their headaches, the Python team will be able to provide a clean modern documentation about package creation and installation, and the world will be a better place ❤️.

If anyone needs help about packaging (for WeasyPrint, pydyf, pyphen, etc.), we’ll definitely take the time to discuss and find clean solutions. I agree that a simple sed is a nice workaround, and that’s a nice way to get everything working for you, for now. But I fear that it will break in future versions, because of setuptools, because of flit, because of Python, who knows… Installing from the wheel is painful, but it’s the way to go, and we’re ready to help anyone to build reliable solutions for reliable and future-proof packaging.

[FelixSchwarz]

Just just pushed WeasyPrint v53 into Fedora rawhide and maybe you can use my spec file to get some hints how to package this. I have to admit that this was not that hard due to the fantastic work of Fedora's Python wranglers.

The Fedora spec is here: https://src.fedoraproject.org/rpms/weasyprint/blob/rawhide/f/weasyprint.spec
The new macros are defined in Fedora's pyproject-rpm-macros RPM (/usr/lib/rpm/macros.d/macros.pyproject).

[liZe]

I have to admit that this was not that hard due to the fantastic work of Fedora's Python wranglers.

Wow. ❤️. Thanks a lot for sharing.

The macros in pyproject-rpm-macros could be really useful for other distributions. But, of course, it would be actually hard to use the same scripts everywhere because of differences between distributions. That’s where the distributions and the Python team will have to work closely together, but it won’t be easy: distributions have different points of view about a lot of things (filesystem paths, package formats…), and they definitely have different points of view about Python packaging too.

Fedora’s workflow uses pip wheel and pip install, with some adjustments made for Fedora’s needs. Most of this code is actually useless for WeasyPrint (as it handles specific setup.py and setup.cfg hacks too), and we can assume that it will become more and more simple with time, now that the format is specified.

I really hope that other distributions will follow this path. Fedora won’t have to change anything when Flit will remove setup.py, because the wheel+install steps will be exactly the same. Distributions relying on another workflow may have to make new adjustments again, just as they did for years with setuptools workarounds. I’m really sorry for them 😢 and hope that they’ll find a clean solution soon.

If you’re a distribution packager, we’re really open to discussion: come talk on our Gitter room, or open a dedicated issue in this repository. We’re closing this issue for now, as a workaround has been given for setuptools users, and some hints have been given for a more reliable packaging.