Please drop setuptools_scm

[ddevault]

This kind of release process is a huge nuisance for those of us packaging software downstream. Not being able to use PyPI tarballs or even GitHub tarballs is requires us to go well out of our way to accomodate your package. Please just conform to normal Python packaging norms.

[FFY00]

Just set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable.

https://github.com/archlinux/svntogit-community/blob/19f4a531a1cc0b1dedc7aa2a2911d79a610b3caf/trunk/PKGBUILD#L18

[ddevault]

I don't think that's a very good workaround for not having good packaging practices.

[FFY00]

Respectfully, I disagree. I understand that it is annoying but that environment variable is there for just that, to override the setuptools_scm version. We could document this better, but overall I don't really see an issue with this approach besides being slightly annoying to packagers, it has absolutely nothing to do with good or bad packaging practices.

[ddevault]

being slightly annoying to packagers

has absolutely nothing to do with good or bad packaging practices

Is this not a contradiction? There's little reason to rock the boat. Just ship with standard Python packaging tools and it'll work better for everyone.

[FFY00]

How is something being slightly annoying to packagers a bad packaging practice??
First of all, "packaging practices" are executed by packagers, not by upstreams. All you can argue is that the upstream is making the packagers' lives a harder, which can be a fair point, but generally is a tradeoff, so it is up to the upstream to choose how much of their workflow they are willing to sacrifice to make your life easier.

Upstreams are free to manage their projects as they want, but I think they should be careful to not make the workflow incompatible, or unreasonable, with popular packaging workflow requirements. This definitely not the case here, I would agree with you if this did actually make it impossible or unreasonable for you to package the project, but that is just not true.

And please note, I am a packager myself. There's little point trying to argue in name of packagers(TM) here.

Just ship with standard Python packaging tools and it'll work better for everyone.

Just wait until you hear about PEP 517....

[ddevault]

Just wait until you hear about PEP 517....

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

And please note, I am a packager myself.

Debian agrees with me:

https://wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

[eli-schwartz]

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

On that note, in search of a solution I wrote to the git mailing list fairly recently: https://public-inbox.org/git/[email protected]/T/

It's highly annoying that github archives are rendered either entirely useless or obscure and difficult to work with, depending on which particular variety of broken setuptools plugin any given project is using. I'll admit that one could do much worse than setuptools_scm, but that still isn't especially ideal IMO.

And PyPI sdist are often not an option as they more often than not don't distribute tests... (This is not a comment on this specific project.)

[FFY00]

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

We do, but things are very complicated.

Debian agrees with me:

wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

That at most only partially agrees with you. The VCS or code repository is not required to be present, it is the one of the resorts when detecting the version.
I understand your annoyance, but it is not really not up to you, or me for that matter, to decide what they are willing to compromise at. As a packager you need to accept when upstream are not willing to make the compromise you are looking for.
If you want to see my position as a lead maintainer, you can check out pypa/build#170. I get your sentiment here, I really do, I just don't think your attitude of "just conform", "this is incompatible" and "it'll work better for everyone" is correct.

[ddevault]

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

pypa/build#170

Funny that you went from -2 to being a part of the problem.

[FFY00]

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

It sure would, but it is up to the maintainers, not you. As a packager I find this solution annoying, but acceptable, I just fix my PKGBUILD and go around my way.

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

I am very aware of the impact of this. It adds up, but it's a one time fix.

Funny that you went from -2 to being a part of the problem.

I am not the main maintainer of this repo, I have only tagged one release, it is not my workflow that this will be breaking.

You can wait for @jaraco to weigh in, but I doubt he'll agree to this, hence my first comment.

[ddevault]

I don't think @jaraco expected to receive pushback for this, and I would hope that the knee-jerk reaction to dismiss these complaints out of hand isn't what we'll get. But @jaraco, please weigh carefully whether or not the perceived benefits of using scmtools-scm, which is almost universally hated among packagers, are worth the trade-off.

[FFY00]

I was never trying to dismiss complaints here, my first comment should have been worded better, sorry. I was trying to provide you an immediate solution, you then escalated saying how this is bad packaging practices.
I am +1 on dropping setuptools_scm, but as I said, it is definitely not my call.

[jaraco]

I appreciate you bringing this concern to bear. Unfortunately, I'll probably need more information to understand what's causing you problems. Since the request here applies to the hundreds of projects I maintain, I'd wish to move it to jaraco/skeleton, where these practices are maintained across those projects, but that's another org, so I can't and I'll just address the concerns here.

Not being able to use PyPI tarballs

You should be able to use PyPI tarballs.

~ $ pip-run --no-binary importlib_resources importlib_resources
Collecting importlib_resources
  Downloading importlib_resources-5.1.0.tar.gz (31 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.1.0-py3-none-any.whl size=24554 sha256=9e6c5526f11d7b45fb761fa597bd2990d973c2dfecfb79bfaffbb7712e654521
  Stored in directory: /Users/jaraco/Library/Caches/pip/wheels/a0/82/c0/6455ddb59a10e136e92e902c63b8864c52c8a8e8146c377766
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.1.0
WARNING: You are using pip version 21.0; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/jaraco/.local/homebrew/opt/[email protected]/bin/python3.9 -m pip install --upgrade pip' command.
Python 3.9.1 (default, Jan  5 2021, 11:10:38) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Under what conditions are you unable to install PyPI tarballs?

Please just conform to normal Python packaging norms.

The setuptools_scm approach is a Python packaging norm that's evolved from best practices that go back to core features in setuptools from 2006 and that have been consolidated in the setuptools_scm under the PyPA umbrella as the blessed approach for leveraging SCM metadata in Python project development. I believe other PyPA projects include this behavior, even built-in (though I'm unfamiliar). I acknowledge that it has some shortcomings (the main one being that Github tarballs don't bring enough metadata), but Github checkouts are still supported:

~ $ pip-run git+https://github.com/python/[email protected] -- -c pass
Collecting git+https://github.com/python/[email protected]
  Cloning https://github.com/python/importlib_resources (to revision v5.0.1) to /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git clone -q https://github.com/python/importlib_resources /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git checkout -q ecc2a24af948c55ecc88f0c3cfc4c3900c8f6246
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.0.1-py3-none-any.whl size=23364 sha256=5fff94d19749d20910aeee76f8ee5e39504e530060591bc3191e1c6b1b5d3b8d
  Stored in directory: /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-ephem-wheel-cache-2oypcf6_/wheels/0f/33/ae/fa8051a526161d3dfd5c2f824a228aea20468ce132c34a3dfa
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.0.1

I'm heavily invested in setuptools_scm and the value it provides, and in my experience, the value far outweighs the limitations it imposes. The benefits include:

• No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.
• Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.
• Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).
• Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.
• There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

Removing setuptools_scm support from this project and the hundreds of others I maintain would add toilsome steps, redundant data, noisy commits, inconsistent states, and reduce the fidelity of the intermediate versions produced.

Can you explain more about what workflows are unmet by the current approach? Perhaps there are other approaches that would satisfy the need.

[FFY00]

Under what conditions are you unable to install PyPI tarballs?

The issue is building from github tarballs, as is common for packagers to do. It is usually desired to fetch from the original source rather than a 3rd party service, but I'll let Drew elaborate on this if he wants.

[ddevault]

Lot to unpack here.

You should be able to use PyPI tarballs.

Kind of, though I ran into issues. Crucially, you can't use the PyPI tarballs combined with a standard Python packaging procedure, nor the GitHub tarballs.

Note as well that no one uses pip for packaging Python stuff downstream. Working around the problem in pip definitely doesn't work for us. For one, we have a hard constraint on not accessing the network during the build process.

No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.

Not that hard. I can offer some advice if you find this difficult.

Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.

Honestly this sounds irresponsible as hell. This leads to broken releases. Surely you've broken a release doing it this way and had to hastily ship a fix? Strong NACK to this as an advantage. I don't want you releasing things into my production systems from the comfort of your bed.

Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).

This can also be dealt with. I have a script for doing releases that deals with this stuff.

Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.

???

There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

No, just downstream workarounds which also have to be maintained.

I don't really want to engage with you on this discussion further. I'm not going to waste my time when I'm up against the effort of getting you to change your approach for "hundreds of projects". I don't think I could possibly make an argument which would overcome your change resistance when the change involved requires that much work.

[jaraco]

Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.

???

Example:

~ $ pip-run -q git+https://github.com/python/importlib_resources@1401cc48e5077088036aa7e729c8995ffbbb9e88 -- -c "from importlib import metadata; print(metadata.version('importlib_resources'))"
5.1.0
~ $ pip-run -q git+https://github.com/python/importlib_resources@3197fe0 -- -c "from importlib import metadata; print(metadata.version('importlib_resources'))"
  WARNING: Did not find branch or tag '3197fe0', assuming revision or ref.
5.0.2.dev4+g3197fe0

Here you can see the commit at 3197fe0 produced an installed version of the prior version (5.0.1) + 0.0.1dev4 (indicating 4 commits past 5.0.1) plus a tag indicating the git commit ID. So any environment with this artifact installed exposes useful information about the intermediate state between releases.

Many (most?) other workflows would have presented as either 5.0.1 or 5.0.2 for that commit, even though it is neither.

[jaraco]

Note as well that no one uses pip for packaging Python stuff downstream. Working around the problem in pip definitely doesn't work for us. For one, we have a hard constraint on not accessing the network during the build process.

To be clear, I was presenting pip as one reference implementation of the PEP 517/518 standards rather than a sole recommended approach. It illustrates that it's possible to use the same standards to produce artifacts for a downstream environment without pip and still honor the standards that allow for tools like setuptools_scm (or other plugins) to be employed and to fully honor zero-network installs (assuming build dependencies are supplied somehow).

I don't really want to engage with you on this discussion further. I'm not going to waste my time when I'm up against the effort of getting you to change your approach for "hundreds of projects". I don't think I could possibly make an argument which would overcome your change resistance when the change involved requires that much work.

I'm glad you appreciate my perspective here. I don't want to have one workflow for a set of projects and a different, more cumbersome workflow for another set of projects based on how the end users wish to consume them. I'd like to have a uniform approach that can be re-used across all of them.

I do think we still have opportunity to employ an approach that meets both of our needs, but if you've given up before even answering my first question, to define what the undesirable behaviors are, and the prescription you offer is to degrade my workflow substantially, then I agree it would be preferable to stick with the status quo.

I welcome you at any point to re-engage in the conversation. I'd be happy to explore options. Once I understand the failing use-cases better, perhaps I'll be persuaded to alter the approach to lessen the downstream burden.

Until then, I'll close this at your request. Best Regards.

[ddevault]

I'm glad you appreciate my perspective here. I don't want to have one workflow for a set of projects and a different, more cumbersome workflow for another set of projects based on how the end users wish to consume them. I'd like to have a uniform approach that can be re-used across all of them.

I don't really appreciate your perspective here, in fact I dis-appreciate it. I instead appreciate the futility of convincing you to change a hundred projects. I think that you should have never used setuptools_scm, for any of those projects, nor should you continue to using them.

Stating this for the record to clarify a misinterpretation of my position, but not to sustain a discussion which, per my clarification, is futile.

[ddevault]

As for why I don't want to engage in a discussion which is probably futile, aside from the normal frustrations of arguing with an immovable object, consider that you just have to defend it N times, and I have to argue for it N^2 times for every project with bad packaging practices. I'll stick to the ones which quickly agree to fix their issues.

[eli-schwartz]

• Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.

That feels like an odd thing to optimize for, e.g. it also has super low friction for creating bad releases that you thought were a good idea at 3AM.
It also guarantees you cannot do code-signing, though I will grant you the python ecosystem has a heavy bias against code-signing and e.g. aggressively removed visibility of it from Warehouse (and IIRC seeks to fully deprecate it?), so this argument may not sway you...

• Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).

IMHO that's not a very compelling argument, since it is just as easy to automate this by maintaining it via a script as it is to maintain it by running the git utility.
Just because it is theoretically possible does not mean it is possible in practice.

• There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

Personally I'd use a ./do_release.py that updates a setup.cfg version = attr: foomod.__version__, consolidates changelogs using something like towncrier (used by pip), commits the results, tags the commit, generates sdist/wheels, uploads the result, and drafts a github release.

You... already do half of this, but you call it tox -e release, delegate version bumping to setuptools_scm, and rely on your own finely handcrafted CHANGES.rst in the PR merging a given feature, then apply git tagging to a commit with the title "Merge pull request #183 from python/feature/90-low-level-reader"

Not only do I not see why "avoid maintaining a development tooling like bump2version" is a bad thing, I think you're already doing it! Even if it is as simple as a series of commands to run you can still do those commands in a python or shell script.

"You say tomato, I say tomato."
("But wait, this is a text-only medium, how can one pointlessly criticize another's pronunciation?" "That's the joke!")

• Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.

This, I don't have an answer for, though. I personally don't believe this is nearly as useful in practice as it is in theory (or that it is worth adding more moving parts for the end user), but who am I to impose that belief on others?

I do think it would be nicer if the tool used to calculate git versions would override the existing version checked into git only when run from a git repo -- that is how non-python projects tend to do such things -- but the existing tools don't seem to support that.

I'd like to draw attention once more to my proposal on the git mailing list: https://public-inbox.org/git/[email protected]/T/

If implemented, this would let git-archive create version files that look like this:

# this will actually be a version string if using a tarball from git-archive, e.g. github tarball downloads
_git_version = '$Format:%(describe)'

if _git_version.startswith('$Format'):
     # updated by release tooling, this is a fallback if enhanced dev metadata is missing
    __version__ = '0.1'
else:
    __version__ = _git_version

And this is already supported by setuptools core version = attr: foomod.__version__ style support.

gitattributes export-subst today only supports identifying direct tag elements, regrettably (see the mailing list discussion) which is the main impediment to actually implementing all this. I hope this will change in the future.

[webknjaz]

@eli-schwartz

It's highly annoying that github archives are rendered either entirely useless or obscure and difficult to work with, depending on which particular variety of broken setuptools plugin any given project is using.

In fact, there's a package that does exactly that — setuptools-scm-git-archive. This project doesn't use it but most of the other setuptools-scm-based libs do. Personally, I do this in all my projects and it works like a charm. @jaraco I recommend you add this into the skeleton, we already have it in the CherryPy repos if you want to see examples.

I'd like to draw attention once more to my proposal on the git mailing list: public-inbox.org/git/[email protected]/T

Just to clarify: setuptools-scm-git-archive does this but in a standardized fashion that's been a de-facto standard for many years already. So it'd be only right to keep using it. Also, many projects have an if-conditional or a try/except detecting the version similarly to what you've described — they attempt to get a specific build version from the metadata (via pkg_resources/importlib_metadata/static file/etc). Please also understand that version = attr: foomod.__version__ is something that is used by setuptools on build (when you run things pip install ... or setup.py sdist) but then in runtime one can grab the version a bit differently which is where you'd normally see those try/excepts.

I do think it would be nicer if the tool used to calculate git versions would override the existing version checked into git only when run from a git repo -- that is how non-python projects tend to do such things -- but the existing tools don't seem to support that.

Well, that's what setuptools-scm basically does. Once the project is installed or built, the metadata is static.

[eli-schwartz]

Thanks, the git-archive plugin does look very neat. I think it would work even better if the disclaimer

Note that it only works for archives of tagged commits (because git currently lacks a format option equivalent to git describe --tags).

got solved via the git mailing list proposal in question.

[webknjaz]

@missa112 that is not related to this repo at all. But I think what you are looking for is https://cryptography.io/en/latest/installation.html#building-cryptography-on-linux. Also, for any further guidance, please ask in the PyCA community channels: https://cryptography.io/en/latest/community.html.

@FFY00 maybe this should be locked...

[FFY00]

Drew, these kinds of comments are not helpful in any way. Please re-evaluate your attitude in this issue, and frankly OSS in general. Remember no-one here is required to give you support or adapt to your needs, maybe you need a refresher on the Apache 2 license... we try to that but it's best-effort and done in our free time, and this kind of behavior honestly only demotivates me, and I am pretty sure the other maintainers too.

With that said, I am sure if someone opens a pull request on https://github.com/jaraco/skeleton introducing setuptools-scm-git-archive, that will help this issue get resolved faster.

[ddevault]

Sorry? I don't understand how what I said was rude or unhelpful or entitled. I was just suggesting that this issue wouldn't be locked on the basis that someone commented in the wrong place.

[FFY00]

This was the culmination of all your behavior during this whole issue, your last comment was the last drop. I am now opting out of this issue, take care.

[ddevault]

Suit yourself. OSS doesn't imply agreement.

[jaraco]

@jaraco I recommend you add this into the skeleton

I am sure if someone opens a pull request on https://github.com/jaraco/skeleton introducing setuptools-scm-git-archive, that will help this issue get resolved faster.

This sounds like a reasonable proposal. I want to investigate what downsides, if any, there are to adopting that feature. The only other thing I'd like to see is an articulation of the motivation behind adopting it - what use-cases does it solve and why aren't those use-cases solved by another preferable technique (using sdists as published on PyPI or a repo clone). I requested this information above, but only got indications of problems downstream, not an actual description of what the problems are. I'm fairly confident there are real problems - I want to be sure to capture what they are and what conditions need to be maintained to alleviate them.

[ssbarnea]

Considering some particular github users are prone to spam projects with their comments, I would have locked the thread long time ago. Lucky I am not a core here.

Such kind of tickets is why I enabled github discussions and moved some tickets there, allowing people to continue to engage in long conversations without adding noise to the issue tracker (or having to lock issues).

[ddevault]

The underhanded personal insults are not called for. I came here with a legitimate interest in the project seeking a solution as a package maintainer. Your comment is out of line. Maybe this thread should be locked if that's what it's going to devolve into.

[eli-schwartz]

@jaraco

At the very least, it's an extra bit of metadata that prevents people from messing up if they use github archives.

I don't know what particular error @ddevault might have hit trying to use the sdist, maybe a traceback would help with forensics on that count... whatever the issue was, it was no doubt frustrating that trying to use github archives instead (a VERY common workaround for packaging problems) did not solve the issue and instead made it worse/more confusing.

So I'd posit that reducing the number of ways a user could end up with a problematic copy of the code, is at least one reason to go ahead with the git archive plugin.

[webknjaz]

reason to go ahead with the git archive plugin.

UPD: setuptools-scm v7+ bundles that plugin's behavior. But the text file template + gitattributes changes are still needed for it to function.