How do I exclude Pip selecting any pre-release?

[notatallshaw]

Description

According to the documentation:

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

• accepting pre-releases for all version specifiers
• excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)

Pip implements the first bullet point with the --pre flag. But I am unsure how to get Pip to follow the second flag, for example if installed "opentelemetry-exporter-prometheus" I will get a pre-release, but I would like it to give an error with some option.

Expected behavior

ERROR: Could not find a version that satisfies the requirement opentelemetry-exporter-prometheus

pip version

23.3.2

Python version

3.11

OS

Linux

How to Reproduce

python -m pip install --dry-run --no-deps --ignore-install "opentelemetry-exporter-prometheus"

Output

Collecting opentelemetry-exporter-prometheus
Using cached opentelemetry_exporter_prometheus-0.43b0-py3-none-any.whl.metadata (1.8 kB)
Using cached opentelemetry_exporter_prometheus-0.43b0-py3-none-any.whl (10 kB)
Would install opentelemetry-exporter-prometheus-0.43b0

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

We don't have the ability to do that, as far as I know. It's a "should", so the spec doesn't require that capability - we could add it as a new feature, but it's not something there has been much interest in until now.

[notatallshaw]

We don't have the ability to do that, as far as I know. It's a "should", so the spec doesn't require that capability - we could add it as a new feature, but it's not something there has been much interest in until now.

Well no part of "Handling of pre-releases" uses a word stronger than "SHOULD" and in previous discussion on handeling of pre-releases you've used the word "MAY" to mean it is required by the spec, and I would consider "MAY" to be weaker than "SHOULD": #12049 (comment)

So it is unclear to me if the spec is only willing to use words like SHOULD or MAY if any specifc line is required at all.

[notatallshaw]

but it's not something there has been much interest in until now

I would be strongly interested in this feature as I don't want pre-releases unexpectedly.

However I think another part of the fact there hasn't been much interest from users is that Pip often doesn't select a pre-release even when according to the spec it reads to me as though it "SHOULD", e.g. #12469

I would make the assumption that if this happened more frequently, then a lot more users would also be interested in not unexpectedly getting pre-releases.

[pfmoore]

So it is unclear to me if the spec is only willing to use words like SHOULD or MAY if any specifc line is required at all.

My reading of the spec is that only the text

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

is required, as only that part of the text avoids the terms "SHOULD" or "MAY". But I'll happily concede that the spec is unclear, if that helps. I certainly don't feel that I have a good understanding of what the "right" behaviour around pre-releases should be (or even whether what the spec says is the best thing to do in practice).

I would be strongly interested in this feature as I don't want pre-releases unexpectedly.

I would support a hard-line stance of never selecting pre-releases under any circumstances unless --pre is explicitly specified. But that's impossible for backward compatibility reasons, and I'm pretty sure it would be considered in violation of the spec, so I don't see that happening.

Anything else is always some form of "do what I mean" approach, and as such is likely to have problematic edge cases at a minimum.

I would make the assumption that if this happened more frequently, then a lot more users would also be interested in not unexpectedly getting pre-releases.

Hmm, to me that argues in favour of leaving pip's behaviour as it is. I certainly don't see any great advantage in implementing stricter conformance to the spec, along with a flag that allows you to (in effect) opt out of that behaviour, while expecting most users to prefer to add the flag...

(We're both responding here and on Discourse - can I suggest we find a single place to have the discussion until we get some form of consensus?)

[potiuk]

Speaking of interest - I wanted to just document a case where more selective --pre flag would be really useful. It's a bit tangential (but I believe very much related).

We would be rather interested in Airflow (to the point of maybe proposing an implementation in pip if that's considered as acceptable) to allow pre-release for only selected packages, not all of them. Currently --pre is a bit too much for us when we run Airflow CI and prepare several packages together.

I know our case is rather niche, but let me explain it, maybe it will be useful for others. Or maybe someone will direct us in finding a better way of doing what we are doing (maybe there are better ways already that I am not aware of).

In Airflow's CI we always build many packges together from our monorepo: airflow + ~90 providers. Sometimes in main version we build packages that will depend on the FUTURE version of core airflow. For example (recently)

a) we prepare apache-airflow-provider-common-io 1.0.0.dev0 that has a required dependency on upcoming version 2.8.0 of apache-airflow
b) in the same build we prepare apache-airflow 2.8.0.dev0 - pre-release

During our CI we want to install both, because we want to run tests with both packages installed as part of the future PROD container image where both are installed.

With the current pip the only way you can install these two together is to add --pre flag with install, otherwise the requirement of apache-airflow>=2.8.0 in apache-airflow-providers-common-io 1.0.0.dev0 will exclude the pre-release 2.8.0.dev0 version of apache-airflow. But the side effect of it is that when I am installing these two together, I will also potentially install pre-release packages of all other dependencies (including transitive ones) and this is very bad idea for stability of the tests and tested PROD images and might cause unrelated issues caused by 3rd-party pre-release packages.

So what I would love to have is to be able to specify for which packages we accept pre-release versions.

BTW. It's not a high priority issue for us and lack of this (optional and niche, I understand) feature is not blocking us. We currently do that by custom pre-processing of all the .dev0 providers to replace apache-airflow>=2.8.0 with apache-airflow>=2.8.0.dev0 which is a poor man's version of selective --pre flag.

BTW. It's very similar story when we have two providers and one of them depends on the "future" version of the other provider - and we solve it in similar way.

[notatallshaw]

Speaking of interest - I wanted to just document a case where more selective --pre flag would be really useful. It's a bit tangential (but I believe very much related).

Tagently to your use case, but of maybe useful context, it's trying to install apache-airflow[all] using a different tool that led me down this rabbit hole of pre-releases: prefix-dev/rip#74 (comment)

[potiuk]

Tagently to your use case, but of maybe useful context, it's trying to install apache-airflow[all] using a different tool that led me down this rabbit hole of pre-releases: prefix-dev/rip#74 (comment)

Very tangently to that one... The all extra for airflow has been long time broken because we have been (ab)-using the way how our setup.py was historically written (and we explained that all should only be used for editable installation (which has been broken in other ways due to changes implemented by pip and the main reason was that what we've done was largely abuse of setuptools and installation process).

So technically what you are trying to do there is not supported by Airflow and if you want to install all extras of Airflow (so far) you really need to use all individual extras - especially if you want to install airflow with constraints - rather than all.

From https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html#bundle-extras :

These are extras that install one or more extras as a bundle. Note that these extras should only be used for “development” version of Airflow - i.e. when Airflow is installed from sources. Because of the way how bundle extras are constructed they might not work when airflow is installed from ‘PyPI`.

Luckily the big change I am about to merge for 2.9 line (and possibly even cherry-pick for 2.8.1 release of Airflow) apache/airflow#36537 - which you were commenting on :) - makes Airflow compatible with PEP-440 PEP-517 PEP-518 PEP-561 PEP-621 PEP-660 PEP-685 in one go and we will hopefully be on the "forefront" of Python packaging - including using Hatchling as build backend and recommending Hatch as front-end.

And a side-effect is that all becomes perfectly valid extra that should be ok to use in both --editable and standard mode.

[notatallshaw]

So technically what you are trying to do there is not supported by Airflow and if you want to install all extras of Airflow (so far) you really need to use all individual extras - especially if you want to install airflow with constraints - rather than all.

Good to know, but my main use case for all is to quickly test if I can break Python package resolvers (mostly Pip as I test and develop different optimizations for it). For actual airflow instances I look after I individually specify extras.

[potiuk]

Good to know, but my main use case for all is to quickly test if I can break Python package resolvers (mostly Pip as I test and develop different optimizations for it). For actual airflow instances I look after I individually specify extras.

Oh yeah. I imagine that our case is a good one to test on - with 650+ "all" dependencies and things like botocore dependency with 100s of applicable versions :). I've spend a lot of time scratching my head trying to solve some of the backtracking issues there. The bad news (for your tests - but good news for our users) is that as part of my PR I bumped a lot of min-versions for a lot of our old dependencies which will make it easier for resolver (i.e. far less number of candidates to consider). But you can still use older versions of Airflow for that.

But even there, I'd recommend to do your tests all the resolve algorithms in a bit different way:

INSTALL_PROVIDERS_FROM_SOURCES="true" `pip install -e .[all]` 

Or equivlent.

This was the canonical way of installing Airflow with all the dependencies - tha was part of the abuse we implemeted BTW. And yes. It's something I implemented years ago to overcome some packaging tool limitations for our case, but lucklly recent PEPS and their implementations in tools like Hatch/hatchling made it possible for us to get rid of that madness.

To be honest I am seriously impressed with the way how the PEPs made it possible (I read all of them + accompanying discussions and I'd say some of the decisions made and especially rejected alternatives were great). It's a marvel of a design - complex on the inside - for good, historical reasons - but very customizable / extensible and rather simple on the outside once you understand the purpose of it and why it's implemented the way it is.

Kudos for the whole PyPA team - it's been hell of a job to get it to that stage I think. Taking into accounts all the complexity of projects like Airflow, it's amasing, we can turn Airflow into a pyproject.toml - only , fully PEP-compatible build system, I think anyone else complaining "I can't move" is just not true.

[notatallshaw]

Small update for anyone who needs to this feature in a pip-like tool, both rip and uv support excluding any pre-releases.