Cannot resolve packages that are split among multiple sources, on 2022.4.8

[ericPrince]

Issue description

If I make a Pipfile with multiple sources, pipenv 2022.4.8 fails for any package that doesn't exist in the first source. Older versions of pipenv including 2022.3.28 worked. This seems like it could be related to #5042.

1. Set up a Pipfile with multiple sources (PyPI and a private source that only contains private packages).
2. Add a package from each source to the Pipfile (aka numpy and a private package that's not on PyPI).
3. Pipenv 2022.4.8 cannot find the private package if PyPI is listed as the first source, or cannot find numpy if the private source is listed first. (pipenv lock fails)

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for my-package

This was not an issue with pipenv 2022.3.28 (tested it).

I will add that specifying which index to use for the packages that come from the 2nd listed source works. However, this gets complex when I have a package from the 2nd listed source (ie an index of corporate packages), and that package has dependencies on both standard python libraries from the first source and on private packages from the 2nd source. I'd really like to not have to re-list every transitive dependency in the Pipfile with the right source to pull it from.

Expected result

Pipenv resolves packages from both pypi and the private source.

Actual result

Pipenv fails to resolve any package that doesn't exist in the first source.

Steps to replicate

(above)

[matteius]

@ericPrince Please see also: #5053 (comment)

[matteius]

Basically the reason that this worked in 2022.3.28 is because index restrictions were only 66% implemented, but didn't apply to the ignore_compatibility_finder which was still as security issue. I am not sure there is a great workaround for this other than to either mirror the pypi package you depend on and have your private pypi as the default index, which is a pip recommendation, or be more explicit in your Pipfile.

[ericPrince]

Sorry for missing that other issue! Gotcha, so this is intended behavior that is newly implemented with 2022.4.8. Thanks for the quick response, and for sharing the relevant pipenv documentation link in your linked comment (https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes)

[matteius]

@ericPrince Yes somewhat intentional, but I can see where maybe future improvements around how to specify the indexes of sub-dependencies might be helpful rather than the current behavior of getting the default index. Say for example you had two private package servers, each with transitive dependencies on the same server -- there is no good way for the non default server to get the transitive dependencies from the secondary index without specifying every requirement and that it should come from that secondary index.

I've thought about this exotic use case, but the problem is how to implement the ability to say transitive dependencies should get the same index as the parent is non-trivial to solve for and may cross into the pipenv-pip boundary for which we want to minimize as much patching of pip as possible.

[ericPrince]

It is interesting to note that pipenv install --skip-lock will successfully find and install packages from the multiple specified indexes, even if pipenv lock can't. It would be really nice to have a feature in pipenv (command line flag or config setting) that lets pipenv try to resolve a package from all sources in order when locking, if the source isn't specified for that package.

It would even be nice if, when you specify a source in the Pipfile, you could specify if it's allowed to be used to find any packages, or if it can only be used for packages that list it as the source. I think that would maintain security of the lock, but allow a user to turn on this convenient but potentially unsafe behavior for sources they trust.