install_search_all_sources = true is not being respected in latest version of pipenv
#5743
Comments
|
In testing that change you referenced, I had to change There was a big regression in the resolver phase when we upgraded to pip 23 because it began searching all sources for packages when it should have only been searching the default source of the Pipfile. Pipenv uses index package restrictions, which is to say that the default index is where things are searched during locking unless not specified in the Pipfile. This has to be this way to prevent package confusion attacks and have deterministic resolution. It sounds like you were benefiting from the bug, by not having to specify things in the Pipfile that were unique to a secondary package index. However as #5736 pointed out, there are edge cases of that, and it basically revealed to me the whole underlying package confusion attack problem was still possible. In that case the torch hashes were different than the other index, and when the install phase happened they could not be validated. Please provide the pipenv commands you are running to further diagnose. |
|
Sorry, this took me some time to get back to. The command I am running is The following example works on the
The When running If I remove the This above failure is consistent with the error I saw in #5543
In this case, we have the sub dependency (gain-marshmallow) in the Pipfile with an explicitly defined index and it's failing regardless. Although it appears that the failure is that pipenv isn't using the 2nd index at install_search_all_sources
Just for my understanding, what are the expected semantics around multiple indexes? My understanding is that
unless I'm misunderstanding, I don't think that case applies to our example because we have the index explicitly defined and the packages in question only exist in one index. Are there still cases where there could be a package confusion attack if the package in question only exists in one index? As an aside, I haven't experienced issues like this with ruby's bundler or javascript's npm/yarn. As a user, I would expect the scenario I have outlined to just work and am surprised at how many issues we have hit over the years with pipenv around private registries. Please let me know if you need any other information or if I can help in any way, I want to ensure that pipenv continues to work for our organization without future issues and am willing to spend my own time helping to make that happen. |
|
@CNDW Can you set intall_search_all_sources to false and relock then try the
This is right for the first 3 points but |
|
Also want to apologize for problems you have had with private indexes -- pipenv has been through a few different maintainers, I've been working with it for going on two years, and we have had to bring the code base through several transformations to get to a more maintainable state. Index package restrictions was one of the first things I took on because making the project more secure is a top goal. I did test the change you referenced prior to releasing it and again more recently with a Google Artifact Registry I setup and here is an example Pipfile/Pipfile.lock that works for me which shows resolution across indexes. I invited you to a slack group in case there is more opportunity to collaborate on figuring out the issue: https://pyslackers.com/web/slack |
|
I did run the command with |
I appreciate your work on all of this and am very grateful to your responsiveness on these issues. I've joined the slack you posted if you want to take further conversations there |
|
I've spent probably too much time troubleshooting this issue from my end with no real info. My key takeaways
I've created a test that approximates the issue I experienced here, however the test passes I've reproduced the scenario using a public gemfury account here along with some stripped down versions of our private dependencies. This fill fail intermittently for different reasons as I tweak the At this point, I need to start focusing my time on making changes on my end to work around @matteius thank you for your time, I'm going to close this issue out so you can focus on other issues. |
|
Thanks for your feedback @CNDW -- we restored the patch to place the active running pipenv in the sys.path for the imports so hopefully 2 is less of an issue now. I am trying to consider if there is a minimal way to further patch the package finder so the resolve can use index restricted packages but we can possibly fix the usage of |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue description
The latest version of pipenv
v2023.6.18has caused our repos to fail install. This is a regression from the solution we found in #5543 - see that issue for more detailed information.Downgrading to
v2023.6.12makes the repos install correctly.The issue appears to be caused by this change in PR #5737
The change effectively nullifies the
install_search_all_sources = truesetting in thePipfileand results in transitive dependencies published on a private registry not being foundExpected result
package installs should be able to locate sub-dependencies when they are not defined in the
Pipfilebutinstall_search_all_sources = trueis set.Actual result
Dependency Installs are failing to find sub-dependencies that are not defined in the
Pipfileand are published on a private registry despite the inclusion ofinstall_search_all_sources = truein thePipfileSteps to replicate
See issue #5543 for more details on reproducing
The text was updated successfully, but these errors were encountered: