Poetry hangs when installing libraries depending on pyparsing on M1 Apple Silicon

[fernandobperezm]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Monterrey 12.4, MacBook Pro (13-inch, M1, 2020)
• Poetry version: 1.1.14
• Link of a Gist with the contents of your pyproject.toml file: gist

Issue

Hello! I recently changed my intel mac with an M1 one.
In the process, I noticed that all my poetry installations hang (more than 900secs without finishing) if any library I'm installing depends on pyparsing (>=2.0.2,!=3.0.5).

The followings are the steps to reproduce my environment.

poetry new pytest-demo
cd pytest-demo 
pyenv local 3.9.7
poetry env use 3.9.7

Then I comment the pytest dependency in the default pyproject.toml file, it should look like the following:

[tool.poetry]
name = "pytest-demo"
version = "0.1.0"
description = ""
authors = ["Fernando Benjamin PEREZ MAURERA <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"

# [tool.poetry.dev-dependencies]
# pytest = "^5.2"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Then I run the following command (using the -vvv) option:

poetry add pytest -vvv -D

The output of this command is the following

Using virtualenv: /Users/fernandobperezm/Library/Caches/pypoetry/virtualenvs/pytest-demo-nBp3JOZu-py3.9
PyPI: 152 packages found for pytest *
Using version ^7.1.2 for pytest

Updating dependencies
Resolving dependencies...
   1: fact: pytest-demo is 0.1.0
   1: derived: pytest-demo
   1: fact: pytest-demo depends on pytest (^7.1.2)
   1: fact: pytest-demo depends on pytest (^7.1.2)
   1: selecting pytest-demo (0.1.0)
   1: derived: pytest (>=7.1.2,<8.0.0)
PyPI: 1 packages found for pytest >=7.1.2,<8.0.0
   1: fact: pytest (7.1.2) depends on attrs (>=19.2.0)
   1: fact: pytest (7.1.2) depends on iniconfig (*)
   1: fact: pytest (7.1.2) depends on packaging (*)
   1: fact: pytest (7.1.2) depends on pluggy (>=0.12,<2.0)
   1: fact: pytest (7.1.2) depends on py (>=1.8.2)
   1: fact: pytest (7.1.2) depends on tomli (>=1.0.0)
   1: fact: pytest (7.1.2) depends on atomicwrites (>=1.0)
   1: fact: pytest (7.1.2) depends on colorama (*)
   1: selecting pytest (7.1.2)
   1: derived: colorama
   1: derived: atomicwrites (>=1.0)
   1: derived: tomli (>=1.0.0)
   1: derived: py (>=1.8.2)
   1: derived: pluggy (>=0.12,<2.0)
   1: derived: packaging
   1: derived: iniconfig
   1: derived: attrs (>=19.2.0)
PyPI: 43 packages found for colorama *
PyPI: 8 packages found for atomicwrites >=1.0
PyPI: 12 packages found for tomli >=1.0.0
PyPI: No release information found for py-0.8.0-alpha2, skipping
PyPI: No release information found for py-0.9.0, skipping
PyPI: No release information found for py-1.4.32.dev1, skipping
PyPI: 4 packages found for py >=1.8.2
PyPI: 4 packages found for pluggy >=0.12,<2.0
PyPI: 39 packages found for packaging *
PyPI: No release information found for iniconfig-0.0, skipping
PyPI: 5 packages found for iniconfig *
PyPI: 9 packages found for attrs >=19.2.0
   1: selecting py (1.11.0)
   1: selecting pluggy (1.0.0)
   1: selecting iniconfig (1.1.1)
   1: selecting attrs (21.4.0)
   1: selecting tomli (2.0.1)
   1: fact: packaging (21.3) depends on pyparsing (>=2.0.2,<3.0.5 || >3.0.5)
   1: selecting packaging (21.3)
   1: derived: pyparsing (>=2.0.2,!=3.0.5)

The installation then hangs and never ends (at least not after 900secs which is the max amount of time I've waited). I don't know if this is something related to my python version (3.9.7). This happened as well with other libraries, such as dask using poetry add "dask[complete]"

If you need more info by running other commands I'm more than happy to help.

[dimbleby]

I've previously seen people report that this goes away after they clear their cache.

However at that point they've cleared their cache and have no means to reproduce the problem and no-one can debug it.

So if you're willing to do a bit of work, it would be lovely if you can

• move your cache somewhere, and see whether that allows things to unstick
• if it does, then gradually reintroduce parts of your moved cache until you can reproduce the issue with as minimal a cache as possible
• then find some way of sharing that - or debug it yourself! - so that someone can investigate

Of course this might not be anything to do with the cache at all, who knows.

[fernandobperezm]

Thanks, David!

I tried deleting and re-creating the cache, and that worked.

I also tried to dissect the problem and found a possible culprit: an _http folder inside poetry's PyPi cache (dir is: <poetry cache dir>/cache/repositories/pypi/_http, in my machine it is located at: /Users/fernandobperezm/Library/Caches/pypoetry/cache/repositories/pypi/_http).

I'm sharing a Google Drive link (here) with two _http folders. The one suffixed with old is the one that causes the issue. The one suffixed with new is one created by poetry and led to successfully installing pytest.

Sadly, I do not know about the structure of this folder or its contents, so I'm not able to debug it myself. Hopefully, somebody may have a look at it. If not, I can take a look at it, but I'll need some guidance on how to do so 😅

[dimbleby]

pretty sure the problem is going to be this lockfile: /a/4/5/1/5/a4515689376b70d90d7a9b3311690220aedc069190a8454969903b1c.lock.

I imagine you hit Ctrl-C or similar at some inopportune moment, and that left your cache locked forever.

Ultimately that file will I think come from filelock via cachecontrol.

filelock is deprecated, but efforts to replace it in cachecontrol are not progressing: psf/cachecontrol#109.

I don't know whether it's even possible to write filelock or equivalent in a way that doesn't sometimes leak an unwanted lockfile - but in view of the lack of recent movement in cachecontrol, we shouldn't expect it to happen any time soon.

Which is to say that I think poetry is likely stuck with this: and will just have to keep recommending that folk clear their cache when they hit it.

[fernandobperezm]

Thanks for the input and clarification, David. Your description makes total sense to me. I was in an airport with a very bad internet connection and probably hit Ctrl+C to restart the network or something similar.

It's good that we found a possible culprit for this issue. Sadly it is not something that poetry/contributors can change/fix anytime soon, but it's 100% understandable why.

I'm happy with the solution clearing cache as it works perfectly. Having said that, I would close the issue. However, I do not know if you guys prefer to have it open to keep track of a (possible?) change in cachecontrol? I give control to you of the issue status 😆

[johnthagen]

Related to:

• Lockfile generation hangs indefinitely when cache is corrupted #5922

[intgr]

I don't know whether it's even possible to write filelock or equivalent in a way that doesn't sometimes leak an unwanted lockfile

Yes. Many Unixes have flock(), which gets automatically unlocked when the file descriptor is closed (e.g. when the process exits or is terminated for any reason). On Windows it's possible to set files to be deleted when their handle is closed.

filelock is deprecated, but efforts to replace it in cachecontrol are not progressing

Nit: cachecontrol uses lockfile, not filelock. lockfile uses some weird hardlink API that requires explicit cleanup, but filelock uses flock internally, which as described above, gets automatically released on exit.

[dimbleby]

should be fixed by cachecontrol 0.12.12

[dimbleby]

ugh, cachecontrol 0.12.12 has been yanked on the grounds that it was breaking, with 0.13.0 promised but not yet released. Fingers crossed...

[dimaqq]

Shall we vendor a fixed version of CacheControl in the meantime?

[dimaqq]

So, ugh, I think I've hit the same issue with poetry==1.2.2 on Linux.
That should have the fix from Sept the 18th, shouldn't it, @neersighted ?

[neersighted]

The fix is in Poetry 1.3 -- we're planning to cut a beta soon, and to release 1.3 very soon after that. I suggest trying an install from Git in the mean time if you'd like to test the fix.

[maknotavailable]

This fixed the issue for me:

poetry cache clear PyPI --all
poetry cache clear _default_cache --all

Thanks to #6526 (comment) and #7076 (comment) !

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.