-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipenv fails to resolve subdependency when viable solution exists #2596
Comments
I've been digging into this a bit. It appears that if you specify a top-level requirement like this then the pip resolver will resolve package dependencies correctly and combine the specifiers regardless of where the package comes from. So in the case above we resolve two requirements: This seems to be similar to #2583 - the underlying issue is that I (and I'm sure others) assume that a requirement like |
It is, indeed. The root issue is that Pipenv’s resolver (which is basically pip-tools) lacks a feature called backtracking. Think of it like save-load points in an RPG video game—you save before you make a decision, and then play on to see what happens. If you don’t like the outcome, load and do things differently, until you get what you want. If nothing you do is satisfying, load an earlier save and try again from there, and so on. Backtracking is not a simple thing to implement. It is not difficult (basically an already-solved problem), but requires significant resources to implement, which the project lacks. Please do try to work on this issue if you’re interested, we will be very, very thankful for your effort. |
Can you elaborate on how backtracking would solve this particular case? I'm very interested in helping if I can - perhaps it's the coffee (or lack thereof) but I cannot see how backtracking fits in specifically here. How would we know when to snapshot? Would this not cause a very exhaustive search which would slow down locking? Wouldn't the number of snapshots and backtracks grow exponentially? Isn't the real problem here that the resolver cannot work out that |
Consider this simple scenario. There are packages A, B, C, and D.
And you specify A in your Pipfile. The resolver can be thought of working in rounds. For each round, we look inside each package to find their dependencies, add them into the list, and repeat, until we cannot find anymore. R0: A. Now let’s consider versions. Say A, B, and D each has one version (to keeps things simple), while C has two, C1 and C2.
R0: A. What Pipenv current does is to always prefer the latest, so… R1: A, B, C2. What backtracking allows, is to save before we made that decision in round 1. R1: A, B, C… save here. Now generalise this. Package X may require P<5 in v1, and P>5 in v2. We might choose v2 first, but if that prooves to be a problem later on (perhaps Y requires P<5 in all versions), we can backtrack and choose the v1 series of X instead. Those conflicting version requirements you see generally come from this kind of more complicated graphs, and backtracking is the solution to it (if there is a solution at all). Backtracking resolution algorithm is largely a solved problem, and there are implementations in various languages. Python, however, is more lacking in this regard. Mixology, which is used by Poetry (another tool that also uses a lock file), and largely similar to Ruby’s Molinillo (used by Bundler and Cocoapods), is the only one I know in real-world production use. Zazo is another, but it is in an even earlier stage. I hope the above clarifies things somewhat. This is not a simple topic to work on (though again, not difficult, because everything is already solved, you only need to implement it with a straight head), but would help Pipenv—and I’dsay, the whole Python comminuty—tremendously, if done correctly. |
Without reading into this too much, would something like https://labix.org/python-constraint help here? |
Just for information this is no longer the case since it was way too slow for Python packaging ecosystem. It works well if all the metadata is available via an API but not otherwise. So I implemented in Poetry another resolver which is lazy (it gets the information only when needed), has conflict resolution and a backtracking algorithm. This led to cutting down extreme cases which would take 30 minutes to less than a minute now. And most of the time it will resolve dependencies in less than 10 seconds. |
@sdispater Thanks for the updated information! May I ask if the implementation currently in use in Poetry is still a variant of same backtracking resolver? I’ve been looking into Mixology (and Milinillo), and the problem to me is that the dependency information for each version is fetched too eagerly when populating a specification. I was under the impression that by implementing possibilities as a lazily-evaluated list would solve most of the problem. Is my impression correct, or would there be other bottlenecks that are impossible to get through? Also—do you plan to extract the new mixology implementation so it can be used in another project? It would greatly benefit not only Pipenv, but also the general Python ecosystem in general (e.g. it would solve a large portion of the long-standing pip problem). @oliland Indeed! Constraint Satification is a generalisation of this particular topic. The library works in a lower level (more abstract, mathematical) than specialised dependency resolvers, but they are all solving the same problem in the end. |
Would you be willing to split this out into a separate package? I would love to help with that, if you're time constrained and agreeable, as it could help a lot of projects in the ecosystem. It seems somewhat silly that the large number of tools that are springing up around this area all have to implement their own version of this, with various deficiencies and trade offs. |
@sdispater if you have plans to isolate mixology I am very interested, I have some implementations that haven't made it into pipenv due to time constraints but that probably is a project where we can collaborate a bit. Is it a full sat solver? Haven't yet seen a good implementation in python |
@uranusjr The resolver has a backtracking algorithm but it's rarely used since the cases where a true conflict occurs are rare.
I agree and that's why I rewrote the resolver completely since it was too slow sometimes to be usable.
I tried that but it didn't lead anywhere, at least with the previous algorithm. It would make this worse since you couldn't check that a package's dependency set had already been seen before and thus ditch the package if it had been, so you would inspect packages that did not warrant being checked, slowing down the process even more. @uranusjr @orf @techalchemy I plan on extract it from Poetry at some point, yes. However, I don't have an ETA yet since right now it's tightly coupled with other parts of the codebase (like a version constraint module). So I have to abstract some of the concepts used internally to make it a trully independent library. |
@sdispater thanks for the clarity on that, appreciate the info! Definitely keep us posted, feel free to let us know if we can help out at all |
@sdispater Would you mind if I try to extract Mixology from Poetry into a standalone repository on my own? I’m trying to work on the resolver logic, and it would be great if I could have something to interface against. I will include appropriate attribution, of course. |
While this issue is being resolved is there a workaround for those of us who want to use pipenv for packages which resolve to unresolvable states? For example I have a pipfile which resolves jupyter to an unresolvable state (most likely because I am using python 2.7 which is stuck on ipython<6). I've specified |
@lukesmurray poetry is an alternative dependency version management tool that serves the same purpose of pipenv, but they claim to have a better dependency resolution algorithm than pipenv. In case you can't afford to way this bug to be fixed, you could use it. Pipenv is already integrated in my deployment process, so I particularly rather wait for this to be fixed. |
@uranusjr I would like to help out with this if at all possible. Would you want the dependency resolution to be built on top of the patched version of pip-tools? |
FWIW, this appears to be a recent regression and downgrading to |
hm interesting @ltm thanks for the tip. That gives us a good place to start looking |
@lukesmurray We are still trying to figure out the best approach. It would be worthwhile to take a look at pip-tools and see if it can be built upon, but we don’t necessarily need to build on it. |
Chiming in to say that @ltm solution of rolling back to |
I can report that I am having a similar issue when trying to install Since there are no restrictions on both of the items, there should be no conflicts between both since A minimal
|
nor |
Is this being worked on? I can't find a related PR for this. |
@gerardo I think it is being worked on in passa project. |
It appears to be the same issue with the
And the graph
|
Encountered another such situation: It's not hindering my ability to use the packages, but there is a warning.
$ pipenv graph
Pipfile
Pipfile.lock
|
@nickdani what worked for me with apache-airflow, was:
So, that worked locally. And, after doing that, I could use the generated Pipfile.lock to re-create the virtual env with the installed airflow packages on another machine, using |
I am having a similar problem installing
Unfortunately When installing I'm using the latest |
@gzagatti I'm having the same issue with Airflow. The issue is that Airflow pins flask-login to 0.2.1 but flask-appbuilder is >= 1.11.1, so that pulls in 1.12.0 which requires flask-login >= 0.3. You can pin click==6.7 and flask-appbuilder==1.11.1 BUT that uncovers the nastiest conflict of all which is not resolvable due to some bug in pip-tools (I think). Nowhere in the dependency graph is Jinja2>=2.10 but for some reason that is picked up by pipenv resolver. Since there is indeed a requirement of Jinja2<2.9.0 we find ourselves completely and utterly stuck... cheers! lol
|
@gzagatti @Mokubyow I've got airflow 1.10 working with something like this and
I also have notes that these were needed:
|
@jvstein. Thanks for the useful recommendation. I can report that it has worked for me. I used |
I have the issue again with |
Me too facing this. |
For the airflow folks, poetry worked for me: https://gist.github.com/jasonnerothin/aaacd3c8aec30bfca991fd04e2847ec9 |
I also got a similar issue today
pytest-html 2.0.0 added a pytest >= 5.0, which ended up failing to lock the Pipfile... When I force pytest-html to go to 1.22.0, it'll work. |
I'll add another example, using Pipenv 2018.11.26 (newest release). These two Pipfile examples fail: [[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
redis = "~=2.10"
rq = ">=0.8"
[requires]
python_version = "3.8" [[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
redis = "~=2.10"
rq = "~=0.8"
[requires]
python_version = "3.8" With this error:
This Pipfile works: [[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
redis = "~=2.10"
rq = "<0.13,>=0.8"
[requires]
python_version = "3.8" I resolved this by hand by looking at the setup.py for rq for every release until found the first one that required With a constraint of Poetry is able to find a solution with the same constraints. I prefer the workflow of Pipenv for application development (Poetry is too library focused). But I've had these |
As of the current master branch, the issue mentioned in the original post here is resolved. @antoncohen the 2nd of your two failing examples works on the current master branch (for me anyway): [[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
redis = "~=2.10"
rq = "~=0.8"
[requires]
python_version = "3.8" The first example fails because pipenv doesn't use backtracking in it's current solver, so it doesn't understand that it needs to potentially look through all of the possible satisfying versions of each root requirement once it does initial pinning -- so, when you told it to stick with the lower version (via the For those of you struggling with I can't say when any of these issues were resolved, only that I thought to have a look recently and was pleasantly surprised. We still have plenty of work to do on this but the situation is dramatically improved. Thanks for your patience all! |
Is there anyway to override the version that has a conflict? I can't find how to set the version of the conflicting package. I tried setting it in the Pipfile but I still get the conflict and I can't generate the lock file. |
This is being worked on here: https://github.com/sarugaku/passa
Issue description
You have two projects:
The
setup.py
looks like this:And the
Pipfile
:From the
base-proj
directory, when you runpipenv install -e ../sub-package/
, you get the following output:How can Pipenv resolve
tensorboard
to'<1.10.0,<1.9.0,>=1.8.0,>=1.9.0'
? It seems it is somehow pulling in the dependencytensorboard
oftensorflow==1.9.0
and1.8.0
, and combining them.Expected result
The
tensorflow
dependency lock in the main project seems to conflict with the one defined in thesetup.py
. Specifically thetensorboard
sub-dependency seems to become immediately unsolvable (seelock --verbose
output below).Actual result
After running
pipenv lock --verbose
:Steps to replicate
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).
$ pipenv --support
Pipenv version:
'2018.7.1'
Pipenv location:
'/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'
Python location:
'/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'
Other Python installations in
PATH
:2.7
:/usr/local/bin/python2.7
2.7
:/usr/local/bin/python2.7
2.7
:/usr/bin/python2.7
3.6
:/Users/tom/.pyenv/shims/python3.6m
3.6
:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python3.6
3.6
:/Users/tom/.pyenv/shims/python3.6
3.6
:/Users/tom/.pyenv/shims/python3.6
3.7
:/usr/local/bin/python3.7m
3.7
:/usr/local/bin/python3.7
3.6.5
:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python
3.6.5
:/Users/tom/.pyenv/shims/python
3.6.5
:/Users/tom/.pyenv/shims/python
2.7.15
:/usr/local/bin/python
2.7.10
:/usr/bin/python
2.7.15
:/usr/local/bin/python2
3.6.5
:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python3
3.6.5
:/Users/tom/.pyenv/shims/python3
3.6.5
:/Users/tom/.pyenv/shims/python3
3.7.0
:/usr/local/bin/python3
PEP 508 Information:
System environment variables:
PATH
TERM_PROGRAM
TERM
SHELL
PIP_PYTHON_PATH
OP_entry_name
TMPDIR
OL_client_secret
Apple_PubSub_Socket_Render
TERM_PROGRAM_VERSION
OP_SESSION
HOMEBREW_AUTO_UPDATE_SECS
TERM_SESSION_ID
LC_ALL
USER
ONFIDO_ROLE
SSH_AUTH_SOCK
__CF_USER_TEXT_ENCODING
nvm_prefix
VIRTUAL_ENV_DISABLE_PROMPT
VIRTUAL_ENV
PIPENV_ACTIVE
PWD
OL_client_id
EDITOR
LANG
ITERM_PROFILE
XPC_FLAGS
OL_region
PYTHONDONTWRITEBYTECODE
XPC_SERVICE_NAME
SHLVL
PYENV_SHELL
HOME
COLORFGBG
ITERM_SESSION_ID
LOGNAME
_OLD_VIRTUAL_PATH
LC_CTYPE
OL_username
__pipenv_fish_initial_pwd
COLORTERM
Pipenv–specific environment variables:
PIPENV_ACTIVE
:1
Debug–specific environment variables:
PATH
:/usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin:/Users/tom/.pyenv/shims:/usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/Users/tom/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS
SHELL
:/usr/local/bin/fish
EDITOR
:nano
LANG
:en_US.UTF-8
PWD
:/Users/tom/projects/pipenv-test/base-proj
VIRTUAL_ENV
:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2
Contents of
Pipfile
('/Users/tom/projects/pipenv-test/base-proj/Pipfile'):Contents of
Pipfile.lock
('/Users/tom/projects/pipenv-test/base-proj/Pipfile.lock'):The text was updated successfully, but these errors were encountered: