PDM Builds up a huge cache over time

[jayceslesar]

• I have searched the issue tracker and believe that this is not a duplicate.

pi@raspberrypi:~$ sudo du -xh / | grep -P "G\t"
1.2G	/home/pi/.cache/pdm/http
1.2G	/home/pi/.cache/pdm
1.9G	/home/pi/.cache
3.6G	/home/pi
3.6G	/home
1.3G	/usr
5.1G	/

Not sure how I can better support but this is over the course of several months. Wondering why it doesnt clear itself out?

[jayceslesar]

Related to #683 ?

[pawamoy]

Why would it clear itself? Genuine question.

[jayceslesar]

Why would it clear itself? Genuine question.

I dont know? Is it normal to build up insanely large caches like this?

[pawamoy]

With enough time, you end up installing multiple versions of many libraries, some of which being quite heavy (torch 900MB!, pydantic 30MB, etc). So I understand the size of the cached packages directory.

However I don't know how the http directory works 😄 I think that's just the HTTP responses to packages downloads that are cached.

[frostming]

I dont know? Is it normal to build up insanely large caches like this?

Why not? Care to look at the size of pip, or yarn, npm if you have installed

[jayceslesar]

I dont know? Is it normal to build up insanely large caches like this?

Why not? Care to look at the size of pip, or yarn, npm if you have installed

pip cache size shows about 1/4 the size there but I guess that is to be expected if most of my installations are happening through pdm?

[jayceslesar]

Regardless -- would probably need to collect some better usage stats to compare...

~/Library/Caches
❯ du -sh pip/
1.0G	pip/

~/Library/Caches
❯ du -sh pdm/
2.7G	pdm/

[jayceslesar]

Unrelated but making a venv from this command seems like a bug?

~/Library/Caches
❯ pdm cache clear
5619 files are removed
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /Users/jslesar/Library/Caches/.venv

[frostming]

Unrelated but making a venv from this command seems like a bug?

~/Library/Caches
❯ pdm cache clear
5619 files are removed
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /Users/jslesar/Library/Caches/.venv

It may be just you, I can't reproduce, do you have any plugins installed?

[jayceslesar]

It may be just you, I can't reproduce, do you have any plugins installed?

What plugins do you mean here? I run starship + iterm in zshell

❯ mkdir projects

❯ cd projects

~/projects
❯ ls

~/projects
❯ pdm cache clear
53 files are removed
python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /Users/jslesar/projects/.venv

[Hnasar]

Regarding the cache: I suppose pypa/pip#3138 and pypa/pipenv#3655 don't clear their cache either.

Regarding creating a virtualenv when it's unnecessary, I think this is a regression with PDM2. I also see on my install (with no plugins) that PDM creates a virtual env any time I clear the cache (I even tried cd /; pdm cache clear)
pdm cache probably doesn't need to create a venv.

[lsmith77]

we are using platform.sh for hosting and we keep going over the build cache size. would be nice if there would be a way to define the maximum space the cache is allowed to use up and then use an LRU algorithm to clear stuff from the cache.

note I think the situation is particularily problematic since we are downloading spaCy models as dependencies which are several GB in size.

[iacore]

Can pdm reuse pip's cache?

[lsmith77]

note in our case I suspect the issue is caused by downloading spacy model dependencies via download URL

    "en-core-web-lg @ https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-3.6.0/en_core_web_lg-3.6.0-py3-none-any.whl",
    "de-core-news-lg @ https://github.com/explosion/spacy-models/releases/download/de_core_news_lg-3.6.0/de_core_news_lg-3.6.0-py3-none-any.whl",

however so those files should not get cached.

$ pdm config install.cache
False

[frostming]

$ pdm config install.cache
False

This is not for download cache.