Enable pip to reuse previously built wheel from cache when installing

[Aloisius]

Pull Request Check List

Resolves: #3439

• Added tests for changed code.

The way poetry calls pip install causes pip to ignore its wheel cache of previously built packages. This patch fixes that by using a direct reference (supported since pip 19.1) pip install --no-deps "package @ artifact/package.tar.gz" instead of pip install --no-deps artifact/package.tar.gz. This causes pip to properly use its resolver to check its wheel cache to see if it had previously built that package before instead of rebuilding it every time.

This can substantially speed up poetry install, especially with CI systems that cache the pip cache directory between calls.

Edit: The previous version of this pull request created a temporary requirements.txt file, however pip can use a direct reference on the command line is cleaner.

[maksbotan]

This PR seems to fix the issue for me, will stick with patched poetry until upstream merges this and makes a release.

[abn]

@Aloisius is there any impact on local path dependencies (sdist + dir) here? ie. do we run the risk of a local path dependency being installed from a cache even when the content at source has changed?

[1ace]

@abn: yes, I believe pip will only look at the metadata to decide whether a cache entry matches, which means that if you change the code without changing the package version pip will not know its cache entry is no longer valid.

That said, what use-case do you have in mind where this is something the user would do?

[abn]

I suspect what I am thinking of are edge cases, and not necesarily best practice scenarios. Nonethe less, scenarios a tool like poetry have to support.

One example that comes to mind based on issues we have dealt with in the past is the case of shared dependencies that are installed via direct path references across multiple projects. A common use case (not something I personally like) is for such dependencies to be mounted via the network. In a lot of such cases metadata is often not updated for changes.

You can end up with a case where;

• Project A and B depends on direct reference to Project C.
• A installs C, C gets a bug fix, B installs C.
• Now B ends up with the old version.

Another case would be of remote direct references (I have a sneaky suspicion pip avoids the caches for these anyway).

Yet another case is where the production version is the main branch on a git repository. These are often also packaged up as package-1.0.0a0 and replaced with every CI run in whatever /simple api repository is being used. 😞

And yet another case is where there are c-extensions and dependent system libraries have changed.

All that said, I do not think these cases should deter this enhancement. Particularly because these seem to only impact cases where folks do not update the metadata with code changes. And workarounds here would be to One way to ensure this is not a blocker is to allow for a --no-cache option to be implemented and used (optionally being able to specify what packages you want to disable cache for). This can potentially happen outside of this issue I reckon, I am happy to do thw leg work for that.

[Aloisius]

The logic in pip's wheel cache is nearly identical to chef.get_cache_directory_for_link().

If the shared dependency is inside of poetry's artifact cache, then since the path includes a hash of the original link url+hash, if either of those change, pip's build cache won't be used since the sdist would be at a different path.

We could also add the hash to the link for the direct install:

pip install varint@file:///home/user/.caches/.../varint-1.0.2.tar.gz#sha1=410df920bb74062081c47386c4e2e455d2655f62

That will prevent pip from reusing its build wheel cache if the hash provided changes and poetry could generate one even if the original link to the sdist did not contain one. I only didn't include this because I wasn't entirely sure if it was necessary.

I wouldn't use direct install paths for things outside the poetry artifact cache or urls that don't have a hash fragment (or VCS @hash). Projects themselves aren't really supposed to be using them. PEP 440 makes it pretty clear it is for tools to use, not humans.

[maksbotan]

@Aloisius hi! did you have any progress with this feature?

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.