Non-wheel, non-cached installation fails in poetry >= 1.4.1 for certain setuptools versions?

[ronlut]

• Poetry version: 1.5.1
• Python version: 3.8.8
• OS version and name: macOS 12.5.1
• pyproject.toml:

[tool.poetry]
name = "issue-8320"
version = "0.1.0"
description = ""
authors = ["Rony Lutsky <[email protected]>"]
readme = "README.md"
packages = [{include = "issue_8320"}]

[tool.poetry.dependencies]
python = "^3.8"
package222 = { version = "0.1.1", source="ronlut" }

[[tool.poetry.source]]
name = "ronlut"
url = "https://pypi.fury.io/ronlut/"
priority = "supplemental"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I encounter a reproducible bug with non-cached, non-wheel archived installations.
I suspect the following PR is when it started.
It happens when the cache is empty (a.k.a no tar.gz and no .whl cached) at all (in CI environments for example).
What happens is (links are to relevant lines in source code in v1.5.1 for example):

1. tar.gz is downloaded
2. tar.gz is converted into a wheel, overriding the original tar.gz archive
3. Trying to populate hashes dict with the original archive (tar.gz) hash, but the file doesn't exist anymore, causing the following error:

5  src/poetry/installation/executor.py:803 in _download_link
       801│ 
       802│         # Use the original archive to provide the correct hash.
     → 803│         self._populate_hashes_dict(original_archive, package)
       804│ 
       805│         return archive

   4  src/poetry/installation/executor.py:809 in _populate_hashes_dict
       807│     def _populate_hashes_dict(self, archive: Path, package: Package) -> None:
       808│         if package.files and archive.name in {f["file"] for f in package.files}:
     → 809│             archive_hash = self._validate_archive_hash(archive, package)
       810│             self._hashes[package.name] = archive_hash
       811│ 

   3  src/poetry/installation/executor.py:814 in _validate_archive_hash
       812│     @staticmethod
       813│     def _validate_archive_hash(archive: Path, package: Package) -> str:
     → 814│         archive_hash: str = "sha256:" + get_file_hash(archive)
       815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
       816│ 

   2  src/poetry/utils/helpers.py:244 in get_file_hash
       242│ def get_file_hash(path: Path, hash_name: str = "sha256") -> str:
       243│     h = hashlib.new(hash_name)
     → 244│     with path.open("rb") as fp:
       245│         for content in iter(lambda: fp.read(io.DEFAULT_BUFFER_SIZE), b""):
       246│             h.update(content)

   1  ~/.pyenv/versions/3.8.8/lib/python3.8/pathlib.py:1221 in open
       1219│         if self._closed:
       1220│             self._raise_closed()
     → 1221│         return io.open(self, mode, buffering, encoding, errors, newline,
       1222│                        opener=self._opener)
       1223│ 

  FileNotFoundError

  [Errno 2] No such file or directory: '/Users/ronlut/Library/Caches/pypoetry/artifacts/42/7d/d9/438df92a60e31c7ad0a69f3daad7bc7d645f0ce467376afad763bfb8d0/<SANITIZED>-7.0.9.tar.gz'

I think the wanted fix would be to be able to convert the tar.gz to a wheel without overriding the tar.gz in the process. That way it won't be downloaded again and again and also it won't break when no cache exists. I couldn't debug it as it goes deeply into some inner scripts running in subprocesses. I can open a PR but I need some guidance in understanding how to approach it and why the .whl is overridden.

@quantum-byte your help would be appreciated too as you are familiar with this area.

Thanks a lot!

[dimbleby]

you forgot to provide a way to reproduce this

future is an example of a package that has only a source distribution available - https://pypi.org/project/future/#files. Installing it works just fine:

$ poetry add future==0.18.3

Updating dependencies
Resolving dependencies... (8.6s)

Package operations: 1 install, 0 updates, 0 removals

  • Installing future (0.18.3)

[ronlut]

@dimbleby thanks, after some more investigation it happens in the following scenario (probably more, but that's my scenario):
the installed package has this block in its pyproject.toml:

[build-system]
requires = ["setuptools~=40.6", "wheel", "poetry>=1.0.5"]

I am not sure why it happens, but in this scenario it overrides the .tar.gz with the .whl file.
I uploaded an example package for that. Steps to reproduce:

poetry new issue-8320
cd issue-8320
poetry source add "ronlut" "https://pypi.fury.io/ronlut/" -p "supplemental"
poetry add "package222==0.1.1" --source="ronlut" -vvv

See contents of the tar.gz here

Changed the title to reflect the actual state of the issue. Updated the issue template to include a reproducing pyproject.toml.
It certainly works in poetry 1.3.2. Doesn't work in newer versions.

Thanks

[dimbleby]

I see a completely different error than the one you have reported, which is that the build fails with a stack ending

    File "/tmp/tmp3n3r2ecz/.venv/lib/python3.10/site-packages/setuptools/depends.py", line 7, in <module>
      from .py33compat import Bytecode
    File "/tmp/tmp3n3r2ecz/.venv/lib/python3.10/site-packages/setuptools/py33compat.py", line 55, in <module>
      unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
  AttributeError: 'HTMLParser' object has no attribute 'unescape'

but that's not what you've reported and I suppose the answer to that is simply to use a newer setuptools.

[dimbleby]

also the above isn't a poetry problem, or anyway not one that anyone is likely to be interested in fixing. Just use newer setuptools!

[dimbleby]

the thing you're reporting looks likely to have been fixed at pypa/setuptools#1745

• setuptools used to start by deleting the contents of the directory in which the wheel was to be created
• now it doesn't

so propose that the solution is again: just use newer setuptools, nothing to do here

[ronlut]

@dimbleby I don't really agree with you it isn't a poetry problem, it was introduced in the PR I mentioned as the behavior changed there.
I do agree that the newer behavior is better, but just important to note that it is a breaking change introduced in that PR. It works in poetry 1.3.2, doesn't work in any newer version.

Will use a new setuptools, but it's not always depends on me, it depends on the package that is being installed.
Closing, thanks anyway.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.