Reliance on flit makes packaging incompatible with Void, Arch and Alpine Linux package systems

[ahesford]

The switch to flit in #352 has left the packaging source tarball without a setup.py. While one can use flit install to install packaging, the flit install command provides a means to choose between installing to the system site-packages (possibly in a virtualenv, by providing a custom path to the python executable) or to a per-user site-packages.

Void Linux xbps-src, Arch Build System and Alpine apk-tools package builders all expect to install Python packages into a specific destination directory that is empty except for the contents of the Python package; the distribution package is derived from the contents of this directory. For setuptools-based projects like packaging<20.5, this is accomplished by passing, e.g., --root $DESTDIR for Void Linux. flit install provides no similar mechanism. The closest one could come would be to construct a virtualenv at the $DESTDIR, then use flit install to populate, and then carefully remove all pieces of the virtualenv from $DESTDIR except for the installed package. It should be self evident that this is a horrible workaround.

If you are use flit build and flit publish to build wheels and source tarballs on PyPI, this problem should take care of itself, because both flit subcommands appear to create a setup.py in the source tarball by default. (Note, however, that the generated setup.py appears incorrect because it imports distutils.core.setup, then uses the install_requires and python_requires keywords that seem to be only recognized by setuptools.setup.) However, the PyPI source tarballs for packaging==20.5, packaging==20.6 and packaging==20.7 do not include setup.py. Thus, you are either creating a source tarball by some other means, or invoking flit build --no-setup-py or flit publish --no-setup-py.

Without a setup.py in the source tarball (either generated by flit build or flit publish, or manually created to parallel the package definition in pyproject.toml), new versions of packaging cannot be properly packaged for Void, Arch, Alpine and, I imagine, several other Linux distributions.

[pradyunsg]

IMO those packaging systems need to update themselves for the fact that PEP 517 is a thing in Python now.

We’re invoking python -m build (source code: https://github.com/pypa/build) for building our distributions and I don’t believe we’re packaging things incorrectly.

[uranusjr]

(Periodic reminder that a wheel installer would provide a straightforward workflow for this.)

[ahesford]

Supporting PEP 517 is fine, and moving away from setuptools or distutils is not ipso facto the issue here. As pointed out in response to takluyver/flit#380, flit really isn't really the problem here. I misunderstood the scope of that project.

Installing from the source tarball is possible using pip, which offers --root and --prefix arguments necessary to prepare a destination directory for production of system packages. Using pip, from a system packaging standpoint, presents a couple of obstacles:

1. By default, pip wants to fetch its own build dependencies. This is bad from a repeatability standpoint (I won't say "reproducability" because that has picked up the connotation of identical outputs that Void, at least, won't promise) because building the same distribution package at different times may pull different versions of build dependencies. This should be avoidable with the --no-build-isolation flag for pip.

2. Using pip with a locally built wheel causes the installation of a direct_url.json that encodes the location of the wheel or the source tree, and pip freeze will list that location rather than a version requirement. This helps repeatability for packages that are truly built and installed locally (provided the original wheel or source tree is preserved), but for system packages, prefering an ephemeral and inaccessible location on some build server to a version string actually discards information and prevents pip freeze from meaningfully describing the system python environment. The workaround here is to remove direct_url.json after installation and before the system package is produced, which is functional but perhaps a bit distasteful.

[FRidh]

In Nixpkgs this change is also causing trouble, but from a different point of view. We install everything from source, and with the change of packaging to use flit we end up with a dependency diagram that is no longer acyclic. Now, in principle this issue already existed before this change because pip and setuptools vendor their dependencies, but then at least downstreams would not have to deal with that.

Note however this issue is resolved as soon as flit uses toml instead of pytoml, but that has not been released yet. I've asked for a release. Until then, we can simply cherry-pick that commit and then things are fine. Thus, no need to change in from my point of view, it is just for your information.

[uranusjr]

Adapting PEP 517 means, essentially, to adapt wheels as the common installation medium. All build tools produce wheels. Pip (in PEP 517 mode) installs wheels. Other tools intending to install Python packages, if they choose to not use pip, would therefore need to understand how to build and install wheels. The wheel format is well-specified, and any tool can implement an interface to perform the process of putting files inside a wheel into any target location.

I would suggest this is a good chance for all the tools mentioned (xbps-src, apk-tools, etc.) to come together and discuss how they should migrate off the implicit dependency against setuptools, and “properly” support Python packages in its specified form. With Python packaging now being increasingly more and more based on standards (instead of specific implementations like setuptools), more and more packages are expected to move off setuptools, and delaying the progress will not make the transition easier down the road. People working on Python packaging generally do not involve too much in alternative tools (since it is enough work to keep eyes on one set of them), and having alternative players in the field would help us understand the challenges and take advice to improve the process early.

[FRidh]

I think that discussion is best kept at https://discuss.python.org/c/packaging.

[ahesford]

@FRidh I'm curious to see how Nixpkgs will handle this update (I see packaging there is still stuck at 20.4, as in Void), since you are even more concerned about reproducibility and will definitely need to freeze the build-time dependencies. Is --no-build-isolation sufficient?

More broadly, there isn't an inherent problem with relying on pip to install wheels as long as those wheels can be reliably produced from source code in a way that supports multiple architectures and cross-compilation. (This is not an issue for the pure-Python packaging, but will definitely become an issue for projects that want to abandon setuptools while still building compiled extensions. Note that flit does not even attempt to solve this problem.)

That pip produces PEP 610 output in direct_url.json without a mechanism to disable this output is a symptom of the real issue here: the Python package ecosystem is indifferent to distribution packaging. Sure, I can remove direct_url.json when producing a distribution package of some Python package; but this shouldn't have to be done. Moving packaging off of setuptools before pip and other build and installation tools fully accommodate the needs of system packaging is a step backward.

The Python community seems to acknowledge that packaging and distribution is a difficult problem, because Python-specific solutions to the packaging problem evolve dramatically. Developing an approach that fits together with distribution package systems, rather than attempting to stand in isolation, seems to offload some of this burden from the Python ecosystem to the distributions that want to bundle this software. Fundamentally, all Python packages should be representable as distribution-specific packages, because they need to fit in a larger dependency graph than the Python ecosystem alone can model.

[FRidh]

@FRidh I'm curious to see how Nixpkgs will handle this update (I see packaging there is still stuck at 20.4, as in Void),

I am updating right now NixOS/nixpkgs#105368. Those updates should land in a week, maybe two, in our unstable channel.

since you are even more concerned about reproducibility and will definitely need to freeze the build-time dependencies. Is --no-build-isolation sufficient?

We declare our dependencies and perform builds in a sandbox. We then indeed pass --no-build-isolation.

To build:

@pythonInterpreter@ -m pip wheel --no-index --no-deps --no-clean --no-build-isolation --wheel-dir dist .

To install:

@pythonInterpreter@ -m pip install -e . --prefix "$tmp_path" \
         --no-build-isolation >&2

Setuptools could still try to install, e.g. when using the deprecated python setup.py test. Before we used sandboxing we tried to capture all those cases. Nowadays sandboxing is the default with Nix, and we don't even bother anymore. Which sometimes is a pity though, e.g. when building with Nix in Docker on GitHub Actions. Anyway, we're getting off-topic.

[eli-schwartz]

IMO those packaging systems need to update themselves for the fact that PEP 517 is a thing in Python now.

@pradyunsg you, better than nearly anyone else alive, know precisely how much of a lie this is.

I get it that it takes time and effort to get https://github.com/pradyunsg/installer/ off the ground, and I don't mean to demand you volunteer "ALL YOUR TIME RIGHT NOW" to make free stuff for people.

But it would be really helpful for the purpose of discussion if you don't carefully miss the point by a country mile and suggest the existence of a tool to build wheels identical to https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl means distros don't need tools to install the darned thing.

You are intimately involved in the existing discussions as to why "those packaging systems" do not want to use pip, and generally cannot, and why PEP 517 only describes one half of a solution previously solved by "use the well-thought-out setuptools interface". I'm sure you're well aware by now that distros aren't helped one bit by PEP 517, which contrarily causes a lot of pain and heartbreak.

It's downright depressing to see this hostile attitude toward distros that are literally asking for one thing and one thing only: do not drop support for the old way of doing things until after a new way is implemented.

Distros are not necessarily experts at writing deep integrations into the guts of python PEPs. But we are very much experts at tying together build systems and system dependencies, then deploying your software to tens of thousands of people.

Making fundamental components of the python packaging ecosystem incapable of being installed using distro-compatible tooling, then performing victim blaming ("it's your fault for not writing your own build system to hook into our buildsystem, we don't care if you can package this, distros suck because they don't PEP 517"), is not exactly what I call welcoming.

Once again, it seems we're back to cavalierly saying "screw you" to distros.

[eli-schwartz]

BTW: python -m build going through the pep517 module, while no doubt capable of producing sdists suitable for consumption by pip install, is not a suitable way to create sdists for uploading to PyPI.
("yay for pep517 standard build_sdist()")

The flit_core module pep517 buildapi (not the flit module + command line tool) uses this: https://github.com/takluyver/flit/blob/6a2a8c6462e49f584941c667b70a6f48a7b3f9ab/flit_core/flit_core/sdist.py#L78

class SdistBuilder:
    """Builds a minimal sdist
    These minimal sdists should work for PEP 517.
    The class is extended in flit.sdist to make a more 'full fat' sdist,
    which is what should normally be published to PyPI.
    """

[FFY00]

(Periodic reminder that a wheel installer would provide a straightforward workflow for this.)

It would in fact solve this, though I do wish all core packaging packages used all the same backend to make bootstrapping easier. Otherwise, we have to manually bootstrap all the PEP517 backends required and then build the packages. It helps that flit_core has no dependencies, but it is still an extra thing to bootstrap. One extra step here plus one extra step there and one extra step in another place in the end get all summed up and make a very real difference.

Unless there are is a logical reason to choose flit, I urge the maintainers to reconsider and move back to setuptools.

I could open a discussion on Discourse to propose choosing one build backend for all core packaging packages.

[dstufft]

I don't think it should be too bad to ensure that the sdist is built with a compatibility shim setup.py. It's not zero cost though, as it means this project ends up on the hook for dealing with support requests that end up stemming from a secondary install mechanism.

That being said, downstream projects are going to have to figure out how to deal with a post 517 world sooner or later, and while I think the ideal situation is that there's some standard tool for helping with that, at the end of the day all of these projects are largely handled by volunteers, so there's a reasonable chance the different moving pieces move at different paces, and that might not be available.

[FFY00]

That being said, downstream projects are going to have to figure out how to deal with a post 517 world sooner or later, and while I think the ideal situation is that there's some standard tool for helping with that, at the end of the day all of these projects are largely handled by volunteers, so there's a reasonable chance the different moving pieces move at different paces, and that might not be available.

The more likely workflow would be using pypa/build, to build the wheels, and pradyunsg/installer, to install them. We still have the issue I presented above when building via PEP517, I think we should make sure all these tools and their dependencies use the same PEP517 backend.
I see no logical reason to use flit here over setuptools, neither the issue nor the PR presented any reasoning, it was just a "let's do it". pypa/packaging does not need anything from flit that setuptools doesn't provide, it was just a matter of personal preference... Which I do not believe is good enough reason to break the current workflows or add more complexity when doing it the PEP517 way. I sincerely do not understand why we are unnecessarily breaking things.

[dstufft]

A quick look suggests the rationale was to remove some amount of burden that setuptools imposes (needing to have MANIFEST.in and the packages kwarg synced and up to date) that the newer tooling makes a lot more streamlined.

I also think that there's value in dogfooding the newer tooling/standards to try and shake out places where we can do better. It's kind of a sad state that due to our position in the ecosystem, we're often times the one least able to take advantage of the improvements we make.

I haven't been very active on this project, so I'm not going to say one way or the other what's going to happen. I just wanted to point out that it's not unreasonable that projects move at different speeds here, and that regardless of what happens, folks should not consider this fully resolved until everyone is capable of handling a fully post 517 workflow.

[pradyunsg]

it seems we're back to cavalierly saying "screw you" to distros.

Huh. Okay... sigh

Well, I just don't have the energy to figure out how to properly respond to this at the moment¹.

I do hope that the work I've been doing over the past few years -- establishing communication channels and having conversations with distro maintainers (during in person events as well as virtually), initiating broader discussions about the underlying issues at play here (like https://discuss.python.org/t/2062 and whatnot), starting and working on https://github.com/pradyunsg/installer/, and lots more things that I don't have the energy to go hunting to link here -- stand to show that I very much actively don't take a "distros suck" approach on these matters. If they don't, well, I don't see how more words in a GitHub comment can change that.

FWIW, as a bit of context perhaps, I'd closed #360 the night before, and triaged this early in the morning. You can look at that user's past issue history for better understanding the kind of interactions I was in the mood for avoiding. Little did I know that I was setting myself up for a differently depressing alternative.

Anyway, I'll be fine with whatever my fellow maintainers decide to do here. Feel free to @ me if there's some plan of action that someone wants my thoughts/opinion on. I'm gonna unsubscribe from this issue and focus on all the other things on my plate right now.

---

¹ I recently relocated to a country on the other side of the planet (yes, in a COVID-19 affected world), there's pip's upcoming 20.3 release that's been taking a lotta work and it's been a really long working Sunday for me today.

[ahesford]

The fact that pip wheel --no-deps --no-build-isolation or python -m build --skip-dependencies --no-isolation both seem to inhibit the tendency to uncontrollably fetch and install dependencies from the Internet seems to address a key concern regarding distribution packaging of packaging. (Of course, build depends on packaging so, there is a dependency cycle here; nevermind that.)

The facts that pip install forces PEP 610 (inappropriately for distribution packages, I contend) and that flit expressly disavows responsibilty for building compiled extensions (and refers to setuptools as the "de facto standard" solution here) tell me that the PEP 517 tooling is not at parity with setuptools. As I said before, I can hack around the PEP 610 issue, and the need to support compiled extensions is irrelevant for the pure-Python packaging. But again, I shouldn't have to hack around the build system. Maybe the installer package obviates this, but that is a work in progress, with no tagged releases, so it is certainly not a solution at the moment.

I understand the value of simplifying package definition from the maintainers' viewpoint. I also appreciate the desire to dogfood these solutions. But isn't there a reasonable argument to be made for preserving the legacy setup.py alongside the new PEP 517 definition until the new---and incomplete---tooling is up to par? Yes, this comes at a nonzero cost, but transitions are never zero-cost propositions.

[FFY00]

A quick look suggests the rationale was to remove some amount of burden that setuptools imposes (needing to have MANIFEST.in and the packages kwarg synced and up to date) that the newer tooling makes a lot more streamlined.

I missed that, sorry.

I still think that is a very small improvement for the usecases it breaks.

[uranusjr]

In a way, breaking the setup.py install workflow can be described as an improvement in itself. PyPA has adviced against it for a very long while, and all the recent packaging improvements are essentially built around this very topic. But alternative architectures are moving away from it very slowly (arguably not at all) because “everything works except yours.”

This creates a very difficult position for every sufficiently popular Python package (I dealt with the same complaints a while ago when I stopped including setup.py in the sdist), who want to migrate soon as adviced to avoid future headaches, but alternative packages put pressure to force them dealing with potential last-minute breakage. The only way to make progress is for package maintainers to speak the desire to migrate in volume, and I am more than glad and fully support that PyPA packages are taking the spearhead.

[FRidh]

It would in fact solve this, though I do wish all core packaging packages used all the same backend to make bootstrapping easier. Otherwise, we have to manually bootstrap all the PEP517 backends required and then build the packages. It helps that flit_core has no dependencies, but it is still an extra thing to bootstrap. One extra step here plus one extra step there and one extra step in another place in the end get all summed up and make a very real difference.

It definitely would make it easy, but it is not absolutely necessary. As long as we can avoid cyclic dependencies it should be fine. I don't know how that is with build however. Because its dependency graph is large compared to pip (which vendors its deps), I have not yet attempted using build in Nixpkgs. If the entire dependency tree can simply be untarred/zipped into an environment for bootstrapping, then everything should be fine. Note in Nixpkgs we build these core packages from the git repos.

Using pip with a locally built wheel causes the installation of a direct_url.json that encodes the location of the wheel or the source tree, and pip freeze will list that location rather than a version requirement. This helps repeatability for packages that are truly built and installed locally (provided the original wheel or source tree is preserved), but for system packages, prefering an ephemeral and inaccessible location on some build server to a version string actually discards information and prevents pip freeze from meaningfully describing the system python environment. The workaround here is to remove direct_url.json after installation and before the system package is produced, which is functional but perhaps a bit distasteful.

Indeed the source location is embedded. Since we build in a sandbox it is still reproducible for us

{"archive_info": {}, "url": "file:///build/py-1.9.0/dist/py-1.9.0-py2.py3-none-any.whl"}

It would still be preferable to remove it considering the possibility of building without a sandbox, so I think we should start doing that in Nixpkgs. Given the file holds no value whatsoever for distros I don't see why it would be an issue for distros to remove it themselves. Its really a minor fix after all.

I understand the value of simplifying package definition from the maintainers' viewpoint. I also appreciate the desire to dogfood these solutions. But isn't there a reasonable argument to be made for preserving the legacy setup.py alongside the new PEP 517 definition until the new---and incomplete---tooling is up to par? Yes, this comes at a nonzero cost, but transitions are never zero-cost propositions.

Which tooling do you refer to here, the PyPA tooling, or the distro tooling? PEP 517 and flit have been around for a while now.

[FFY00]

It definitely would make it easy, but it is not absolutely necessary. As long as we can avoid cyclic dependencies it should be fine. I don't know how that is with build however. Because its dependency graph is large compared to pip (which vendors its deps), I have not yet attempted using build in Nixpkgs. If the entire dependency tree can simply be untarred/zipped into an environment for bootstrapping, then everything should be fine. Note in Nixpkgs we build these core packages from the git repos.

If you are able to use a pip with vendored dependencies to install packages then you don't have an issue. Other distros have guidelines that force them to devendor pip and/or make pip unsuitable for installing packages.

[brettcannon]

OK, a lot to unpack here.

First, the tone of this issue needs to change. For instance, claiming that someone "know[s] precisely how much of a lie" something is when it's a subjective view is not in any way helpful or motivating. We are all volunteers here and coming in agitated doesn't help when we are being asked to potentially rollback something that benefits those of us maintaining this project.

Second, I know from a user's perspective this change wasn't useful, but as a project maintainer we benefit as outlined by Donald. And maintaining this project is not exactly easy when it's so far down the dependency graph in the Python packaging ecosystem which is obviously vast as shown by this issue. Had we known things would break for folks we probably would have either advertised it more or gotten feedback on timelines for when we could safely make such a change.

Third, #340 was open for over a month before I landed the PR. Now that isn't to say I expect all users to monitor our issue tracker or anything, but I do want to be clear this was communicated publicly as coming using normal development processes. I do apologize for forgetting it in the changelog, but to me it was infrastructure since we are PEP 517 compatible and thus didn't require an entry so I just didn't think about it.

Fourth, there seems to be some people saying there is some tooling missing before some Linux distros can rely on PEP 517. OK, then can someone please outline the blockers at https://discuss.python.org/c/packaging/14 so we can figure out what is necessary as a community? If it isn't communicated what needs to be prioritized from the Linux distro perspective then there won't be any improvements to the situation to help the Linux distros make the transition to EP 517 which is where things are definitely heading.

Fifth, if the discussion on discuss.python.org reaches a clear conclusion of what's needed and we are willing to accept a user-contributed PR to rollback (no promises as I can't speak for the whole team), what promises on timeline do people have on transferring their tooling over to support PEP 517? While the accusation that "we're back to cavalierly saying "screw you" to distros" has been made, the reverse that we are being asked to be beholden to distro timelines and tooling can also be made. So if we are going to try and make everything work for everyone I want to make sure we are all working together to unblock everyone, which means not only rolling things back for now, but making sure it truly is "for now" and not indefinite.

[ahesford]

At this point, I will probably just do the Void update with pip in the manner outlined by @FRidh. My main issues were:

1. It was not obvious to me that pip would handle installation. Although I've been using Python extensively for scientific computing for 10 years, I have minimal knowledge of package distribution and building outside of setuptools, especially because the current crop of PEP 517 tools ignore the problem of compiled extensions and I rely on those capabilities.
2. The default behavior of pip seems to assume that it has total control of the Python packaging ecosystem, whether installing in system locations, user locations or virtual environments. It silently fetches dependencies and does a lot to hide control of the build process. These things are not appropriate for distribution-level packaging, and the options to disable them (like --no-build-isolation) were not obvious prior to this discussion. It is not clear from its default output that pip constructs its own build environment even when build dependencies are available in the regular environment. setuptools also tries to satisfy its build dependencies, but: a) those dependencies are considered "satisfied" if they are already installed, and b) when pip is unavailable to the build environment, setuptools falls back to easy_install and complains loudly, so it is obvious to identify and correct this behavior.

Now that I know these things, I can adapt.

[di]

This is perhaps slightly off-topic, but this change seems to have also removed the ability for contributors to work on this via editable installs. I get:

$ pip install -e .
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /Users/dustiningram/git/pypa/packaging
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

Is there an alternative workflow I should be using instead? AFAIK PEP 517 doesn't currently support editable installs.

[uranusjr]

Flit’s equivalent to setup.py develop is flit install -s . (or --pth-file if symlinks are not possible).

[FFY00]

You can use PYTHONPATH=. or, if you want to put this in your environment, you can do create a .pth file pwd >my_env/lib/python3.8/site-packages/packaging.pth.

[FFY00]

Or do what @uranusjr just described 😛 Apparently, flit can put symlinks or a .pth file in place for you.

[brettcannon]

Thanks to @FFY00 being willing to help get 'installer' finished, I support reverting back to setuptools and then adding a pyproject.toml file for it. I don't think the flit PR should be blindly reverted, though, as I moved things over to build to make the release process a bit more generic and ditched __about__.py, so the PR will require a little bit of work (I'll see if I can do this in the next week, but if I don't then someone can feel free to pick this up to get it done sooner).