Potential Regression from v2023.6.12 for private packages in transitive dependencies

[justin-yan]

Issue description

In my testing for this comment (#4706 (comment)), I discovered that starting in pipenv==2023.6.18 private packages defined as a transitive dependency (e.g. https://github.com/justin-yan/pipenvtest/blob/b949c571ead9b9a78aed81465ddcc6bedef4c22f/pyproject.toml#L11) are no longer discovered when you have a second index defined (e.g. https://github.com/justin-yan/pipenvtest/blob/b949c571ead9b9a78aed81465ddcc6bedef4c22f/Pipfile#L6)

The linked commit should be a reproduction example, if you replace the source URL with a private index, and then replace the package name with a private package. If it would be helpful, I can give a maintainer access to a private index of mine with a toy package for debugging.

Expected result

Running pipenv lock should result in a lockfile where privatepkg is a dependency.

This works on pipenv==2023.6.12

Actual result

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for privatepkg

Steps to replicate

Install pipenv==2023.6.18, update the Pipfile in https://github.com/justin-yan/pipenvtest/blob/b949c571ead9b9a78aed81465ddcc6bedef4c22f with a private registry, and then run pipenv lock

Conflicting Requirements

I just wanted to note that I read through the changelog (v2023.6.12...v2023.6.18) and this issue (#5736), and I suspect that this is maybe working "as intended", so I wanted to describe my use case a bit more.

In my internal private index, I have packages A and B, as well as service C.

• B has A as a dependency, but also has some public dependencies such as requests, etc.
• C has B as a dependency

I would like for C to be able to declare B as a dependency without also having to declare A as a dependency. One way I can definitely accomplish this is to proxy all packages through my private index and use that as my only index (and if that's the recommended path, please feel free to close this issue), but I just wanted to raise this issue in case this is something that can be supported (since without this, it doesn't feel like "multiple indexes" is a feature that scales very well beyond a small number of private packages).

---

Thanks for your time and consideration!

[matteius]

@justin-yan I believe this is expected behavior with index restricted packages -- the default index (first index of the Pipfile) is used to resolve and install everything that is not explicitly called out in the Pipfile to use an alternative index.

One way I can definitely accomplish this is to proxy all packages through my private index and use that as my only index

That is what we recommend in our docs, or alternatively specifying everything that should come from the private index in the Pipfile. The docs could use some love and improvements, but it does talk about this: https://pipenv.pypa.io/en/latest/indexes/

I would like for C to be able to declare B as a dependency without also having to declare A as a dependency.

Yes, I think that is the long term goal. There are some open discussions within pip about how pip wishes to solve the index restriction problem as well, because basically users that are unaware of package confusion attacks are the most vulnerable and so its really on the tool maintainers to prevent their possibility of occurrence. That means being more strict with how indexes are specified now.

I can imagine some pyproject.toml directive that says pull package X, Y and Z from private index, but it still has to be specified somewhere, and it will prove complicated to marry those index recommendations back up to how pipenv is tracking the indexes is the Pipfile/lock file. Challenging but not impossible, but that is to say requirementslib needs a lot more refactoring to become more maintainable before cool new things like that should be added, and also I think that having pip solidify their direction is important so that we don't go counter to that.