index restricted package cannot find sub-dependency when its index is non-default

[gitpushdashf]

When upgrading from 2022.3.28 to 2022.4.8, a pre-release package identified with ==x.y.z.alpha is no longer installing. I'm not using --pre or anything like that. Going back to 2022.3.28, it installs fine. Or rather, pipenv update --dev works fine.

[matteius]

@gitpushdashf Can you provide more details such as the Pipfile? It has long been true that you need to use --pre in order to resolve and install pre-releases. I cannot think of what would have changed related to this between 2022.3.28 and 2022.4.8 based solely on what you've provided for details so far.

[gitpushdashf]

--pre has never been mandatory for pre-releases, for as long as I've used pipenv, provided you use ==. For instance, I was doing black = "==21.12b0 until the black 22.1 stable release came out.

I'll try to come up with a minimal test case here. Thank you!

Edit: Here's a black example. This is working for me with 2022.3.28 and 2022.4.8, though.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
black = "==21.12b0"

[dev-packages]

[requires]
python_version = "3.9"

[matteius]

@gitpushdashf Ok thanks, I didn't know that it wasn't required to specify --pre when the version is pinned specifically to a pre-release. That being said, your example worked fine on 2022.4.8 for me ...

[gitpushdashf]

Thanks for testing it! I'm trying to reproduce this in a smaller repository and am not having a lot of luck. It seems to be related to using a private pypi, possibly with pre-release packages. I'll let you know if I come up with something.

[matteius]

@gitpushdashf there are recent changes around private indexes, but they didn't first take effect in 2022.4.8. Likely you need to pin your private non-default package by index name in order for it to continue working for you. See this other thread for more details: #4672 (comment)

[gitpushdashf]

I am using a pinned index for private packages. Will keep trying to reproduce.

Actually... Let me check something.

[gitpushdashf]

My main internal package looks like this:

Pipfile that does an editable install of ., with a setup.cfg with dependencies (some internal, some not).

[packages]
# Runtime dependencies are specified in setup.cfg
xyz = {path = ".", editable = true}

[dev-packages]
mypy = "~=0.942"
pytest = "~=6.2"
pytest-cov = "~=3.0"
pytest-httpserver = "~=1.0"
pytest-mock = "~=3.6"
grpc-stubs = "~=1.24"

setup.cfg:

[metadata]
name = private
version = 1.2.3

[options]
packages = find:
install_requires =
    click~=8.0
    cryptography>=3.4
    thisoneisprivate==0.0.7  # This one is private
    ...

This has actually been working, even with 2022.3.8.

I just replaced xyz = {path = ".", editable = true} with xyz = {path = ".", editable = true, index="myindex"}, and I'm getting the same error:

private % pipenv update --dev
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement another-private-package=
=0.0.0a3 (from versions: none)
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 822, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 771, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/utils/resolver.py", line 1066, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/utils/resolver.py", line 862, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.9/site-packages/pipenv/utils/resolver.py", line 663, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for another-private-package==0.0.0a3

Where I install this private package in other repositories, I've been doing private = {version="~=1.2", index="myindex"}

[matteius]

@gitpushdashf you shouldn't have to specify any index for a locally installed editable package since its referencing to install something locally. Are you saying if you use --pre though that it does work, since a pre-release is involved in the dependency chain?

[gitpushdashf]

I just tried --pre and it did not work, either. Same error.

Are you saying that editable installs resolve through whatever index pip is configured to use?

[matteius]

@gitpushdashf No, I am not saying that. All of the editable installs shouldn't resolve through any index because they are locally available packages that are being installed. Don't read too much into that though, because the index restrictions work was all done in prior march releases which you say are working.

In some ways there are not so many changes in the 2022.4.8 release, but in other ways the diff is massive because I split the utils.py into a utils module so I am unfortunately not able to pin point the source of your problem at this time. You can take a look at the diff on github between those two tags and see if anything stands out to you, but I ultimately tried to keep parity and just move the utils around. I'll give it some more thought and get back to you if I think of something.

[gitpushdashf]

Thank you!

I did manage to reduce this down into a smaller test case.

pipenv-5042 % cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${USER}:${PASS}@myindex.internal/pypi/pypi/simple"
verify_ssl = true
name = "myindex"

[packages]
internal = {path = ".", editable = true, index="myindex"}

[dev-packages]
mypy = "*"
pipenv-5042 % cat setup.cfg
[metadata]
name = internal
version = 0.0.1

[options]
install_requires =
    minio~=7.1
    internal-package-only-in-private-repo~=0.0.7
pipenv-5042 % cat pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[gitpushdashf]

Just found out that it works for me if I remove the pypi index. For me, this is an acceptable solution.

Thank you for your help!

[matteius]

@gitpushdashf hmmm I think that I may have introduced this regression with my changes here: https://github.com/pypa/pipenv/pull/5024/files

I will think more about it, please leave this issue opened.

[matteius]

Just found out that it works for me if I remove the pypi index. For me, this is an acceptable solution.

It's how I refactored that messy method, because it detects pypi is in your sources and I moved the link hash generation to after the pypi checking for some reason, had to do with private hash index collection for torch which also exists in pypi -- but now I need to revisit it based on your example. One problem is I may have with testing your example case is actually running the private pypi server. But I am sure that whats happening is your package isn't in pypi and the hash collection is failing on that because one of your sources is pypi.

https://github.com/pypa/pipenv/pull/5024/files#diff-62e85156fc3b274046cc03dabfe6e3ceead503f330b98fb3611645216b4b6e1fL725-R768

[matteius]

Though I am not 100% sure because in your stack trace its happening in the resolver and not actually when its collecting the hashes.

[matteius]

@gitpushdashf Are you 100% sure that your latest example worked in 2022.3.28? I am thinking its not a bug of the hash collection after all, but rather its caused by the more strict index restrictions because your editable package is requiring another package from your private server, but its not called out in the Pipfile which index that secondary package is restricted to, so it infers its the default index which is Pypi. Rather than remove Pypi -- can you swap the order of the sources so that the private one comes first, and then on your pypi package mypy call out that it can use the pypi index?

[gitpushdashf]

Yes, as soon as I switch to 2022.3.28, it works fine as-is.

With 2022.4.8:

If I reorder the sources and put pypi before myindex, it works fine.

If I remove the pypi index, it works fine.

If I add back in the pypi index at the top, it fails.

[matteius]

@gitpushdashf Do you mean if you put pypi after myindex it works fine?

If I reorder the sources and put pypi before myindex, it works fine.

Because later you say:

If I add back in the pypi index at the top, it fails.

I think this all makes sense to me other than how it would have worked in 2022.3.28. Is it the resolution/lock phase that fails for you, or the sync/install phase or both? Note: pipenv install also runs locks so what I really mean for the second part is pipenv sync.

[gitpushdashf]

Yes, that's right. It's defintely the locking phase, re-generating the lockfile. I usually induce that via pipenv install.

[matteius]

@gitpushdashf I thought more about it and I think what happened is the index restrictions were extended in 2022.4.8 release to include also the index restrictions in the ignore_compatibility_finder which didn't have them prior. I suspect that your example it is relying on the ignore_compatibility_finder to resolve the subdependency of the internal package which is now using index restrictions where it hadn't prior, in order to correctly locate packages https://github.com/pypa/pipenv/pull/5024/files#diff-62e85156fc3b274046cc03dabfe6e3ceead503f330b98fb3611645216b4b6e1fR563-R582

I can imagine a case where someone has two private package servers and both package servers have a package that is dependent on another package that is only in that private package server. It would be an edge case of index restrictions today that this would not work as both sub-dependencies would try to resolve to the parent index, but perhaps it would work if the user explicitly called out the sub-dependency and its required non-default index in the Pipfile. Either way I think that particular edge case would be a hard one to account for, despite that on some level it makes sense that the sub-dependency should use the same index the parent dependency is pinned too, in reality the resolver code I think looses this level of insight and it would be hard to make that mapping connection in the code.

[matteius]

Though my logging doesn't seem to indicate that its using ignore_compatibility_finder at all during your example. Not 100% how this could have just cropped up in the 4.8.2022 release. In thinking about if there would be value in trying to pin the sub-requirements to the same index as the parent I think it would break more cases where the private package's sub-dependency are public pypi packages. I'm going to let this issue rest for now, and give it a potential bug label and a new title, though I am not sure that a lot can be done to support such an exotic example that I've come up with. At least the original case has a resolution.

[matteius]

Ok, I am going to treat this as a question/request for support on the current behavior. If we want to improve the sub-dependency resolution of index restricted packages, then we need someone to write up an enhancement request/proposal that talks through a logical way to do that, or at least captures what the intended requirements are. Thanks!