Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using venv creator creates broken environment in Ubuntu 22.04 #2340

Closed
icezyclon opened this issue May 6, 2022 · 19 comments · Fixed by #2415
Closed

Using venv creator creates broken environment in Ubuntu 22.04 #2340

icezyclon opened this issue May 6, 2022 · 19 comments · Fixed by #2415

Comments

@icezyclon
Copy link

Issue

Using venv as creator - virtualenv --creator=venv myenv - creates a broken virtual environment on Ubuntu 22.04.
It produces two different activate scripts, one in myenv/bin and one in myenv/local/bin.
Depending on which one I activate, either it uses the wrong Python Interpreter or the wrong Pip:

>>> source myenv/bin/activate 
>>> which python3 && which pip3
/home/loki/myenv/bin/python3
/home/loki/.local/bin/pip3
>>> deactivate
>>> source myenv/local/bin/activate
>>> which python3 && which pip3
/usr/bin/python3
/home/loki/myenv/local/bin/pip3

I found this post regarding a change in the sysconfig.get_default_scheme() for Ubuntu 22.04, which might be the reason why virtualenv tries to install into myenv/local/bin instead if just myenv/bin.

Environment

Provide at least:

  • OS: Ubuntu 22.04 LTS x86_64
  • pip list of the host python where virtualenv is installed:
Package                 Version
----------------------- --------------
apturl                  0.5.2
asn1crypto              1.4.0
bcrypt                  3.2.0
beautifulsoup4          4.10.0
blinker                 1.4
Brlapi                  0.8.3
certifi                 2020.6.20
chardet                 4.0.0
click                   8.0.3
colorama                0.4.4
command-not-found       0.3
cryptography            3.4.8
cupshelpers             1.0
dbus-python             1.2.18
defer                   1.0.6
distlib                 0.3.4
distro                  1.7.0
distro-info             1.1build1
dnspython               2.1.0
duplicity               0.8.21
evdev                   1.4.0
fasteners               0.14.1
filelock                3.6.0
future                  0.18.2
gitinspector            0.4.4
gpg                     1.16.0-unknown
html5lib                1.1
httplib2                0.20.2
idna                    3.3
importlib-metadata      4.6.4
jeepney                 0.7.1
keyring                 23.5.0
language-selector       0.1
launchpadlib            1.10.16
lazr.restfulclient      0.14.4
lazr.uri                1.0.6
lockfile                0.12.2
louis                   3.20.0
lxml                    4.8.0
macaroonbakery          1.3.1
Mako                    1.1.3
Markdown                3.3.6
MarkupSafe              2.0.1
mechanize               0.4.7
menulibre               2.2.2
monotonic               1.6
more-itertools          8.10.0
netifaces               0.11.0
numpy                   1.21.5
oauthlib                3.2.0
olefile                 0.46
paramiko                2.9.3
pbr                     5.9.0
pexpect                 4.8.0
Pillow                  9.0.1
pip                     22.0.4
platformdirs            2.5.1
powerline-status        2.8.2
protobuf                3.12.4
psutil                  5.9.0
ptyprocess              0.7.0
pycairo                 1.20.1
pycups                  2.0.1
Pygments                2.11.2
PyGObject               3.42.0
PyJWT                   2.3.0
pymacaroons             0.13.0
PyNaCl                  1.5.0
pyparsing               2.4.7
pyRFC3339               1.1
python-apt              2.3.0+ubuntu2
python-dateutil         2.8.1
python-debian           0.1.43ubuntu1
python-magic            0.4.24
pytz                    2022.1
pyxdg                   0.27
PyYAML                  5.4.1
reportlab               3.6.8
requests                2.25.1
requests-toolbelt       0.9.1
scour                   0.38.2
screen-resolution-extra 0.0.0
SecretStorage           3.3.1
setproctitle            1.2.2
setuptools              62.1.0
six                     1.16.0
soupsieve               2.3.1
sshuttle                1.0.5
stevedore               3.5.0
systemd-python          234
ubuntu-advantage-tools  27.8
ubuntu-drivers-common   0.0.0
ufw                     0.36.1
unattended-upgrades     0.1
urllib3                 1.26.5
usb-creator             0.3.7
vboxapi                 1.0
virtualenv              20.14.1
virtualenv-clone        0.5.7
virtualenvwrapper       4.8.4
wadllib                 1.3.6
webencodings            0.5.1
wheel                   0.37.1
xdg                     5
xkit                    0.0.0
zipp                    1.0.0

Output of the virtual environment creation

virtualenv --creator=venv myenv -vvv --with-traceback

287 setup logging to NOTSET [DEBUG report:39]
291 find interpreter for spec PythonSpec(path=/usr/bin/python3) [INFO builtin:62]
291 proposed PythonInfo(spec=CPython3.10.4.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:69]
291 accepted PythonInfo(spec=CPython3.10.4.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:71]
292 filesystem is case-sensitive [DEBUG info:29]
308 create virtual environment via Venv(dest=/home/loki/myenv, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix) [INFO session:52]
309 create folder /home/loki/myenv/local/lib/python3.10/dist-packages [DEBUG _sync:24]
309 create virtualenv import hook file /home/loki/myenv/local/lib/python3.10/dist-packages/_virtualenv.pth [DEBUG api:95]
309 create /home/loki/myenv/local/lib/python3.10/dist-packages/_virtualenv.py [DEBUG api:98]
309 ============================== target debug ============================== [DEBUG session:54]
310 debug via /home/loki/myenv/local/bin/python /home/loki/.local/lib/python3.10/site-packages/virtualenv/create/debug.py [DEBUG creator:224]
309 {
  "out": "",
  "err": "No such file or directory",
  "returncode": 2,
  "exception": "SyntaxError('invalid syntax', ('<unknown>', 0, 0, '', 0, 0))"
} [DEBUG session:55]
310 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/loki/.local/share/virtualenv) [INFO session:59]
311 got embed update of distribution pip from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [DEBUG via_disk_folder:135]
312 got embed update of distribution wheel from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [DEBUG via_disk_folder:135]
313 got embed update of distribution setuptools from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [DEBUG via_disk_folder:135]
314 got embed update of distribution setuptools from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [DEBUG via_disk_folder:135]
314 got embed update of distribution pip from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [DEBUG via_disk_folder:135]
314 got embed update of distribution wheel from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [DEBUG via_disk_folder:135]
315 install setuptools from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/setuptools-62.1.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
315 install pip from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
315 Attempting to acquire lock 140460470067568 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:169]
315 Attempting to acquire lock 140460469877152 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:169]
315 Lock 140460470067568 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:173]
315 Lock 140460469877152 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:173]
315 install wheel from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.37.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
316 Attempting to release lock 140460470067568 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:203]
316 Lock 140460470067568 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:206]
316 Attempting to acquire lock 140460467461504 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:169]
316 Attempting to release lock 140460469877152 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:203]
316 Lock 140460467461504 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:173]
316 Lock 140460469877152 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:206]
316 Attempting to release lock 140460467461504 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:203]
316 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip-22.0.4.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/pip-22.0.4.dist-info [DEBUG _sync:51]
316 Lock 140460467461504 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:206]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools-62.1.0.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools-62.1.0.virtualenv [DEBUG _sync:51]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel-0.37.1.virtualenv [DEBUG _sync:51]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/distutils-precedence.pth to /home/loki/myenv/local/lib/python3.10/dist-packages/distutils-precedence.pth [DEBUG _sync:51]
317 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel [DEBUG _sync:51]
318 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/_distutils_hack to /home/loki/myenv/local/lib/python3.10/dist-packages/_distutils_hack [DEBUG _sync:51]
319 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools [DEBUG _sync:51]
319 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip to /home/loki/myenv/local/lib/python3.10/dist-packages/pip [DEBUG _sync:51]
322 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel-0.37.1.dist-info [DEBUG _sync:51]
326 generated console scripts wheel3.10 wheel3 wheel-3.10 wheel [DEBUG base:45]
347 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/pkg_resources to /home/loki/myenv/local/lib/python3.10/dist-packages/pkg_resources [DEBUG _sync:51]
352 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools-62.1.0.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools-62.1.0.dist-info [DEBUG _sync:51]
354 generated console scripts  [DEBUG base:45]
374 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip-22.0.4.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/pip-22.0.4.virtualenv [DEBUG _sync:51]
375 generated console scripts pip3 pip pip3.10 pip-3.10 [DEBUG base:45]
375 add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [INFO session:64]
376 write /home/loki/myenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
376     home = /usr/bin [DEBUG pyenv_cfg:38]
376     implementation = CPython [DEBUG pyenv_cfg:38]
376     version_info = 3.10.4.final.0 [DEBUG pyenv_cfg:38]
376     virtualenv = 20.14.1 [DEBUG pyenv_cfg:38]
376     include-system-site-packages = false [DEBUG pyenv_cfg:38]
376     version = 3.10.4 [DEBUG pyenv_cfg:38]
376 created virtual environment CPython3.10.4.final.0-64 in 90ms
  creator Venv(dest=/home/loki/myenv, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/loki/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]
@icezyclon icezyclon added the bug label May 6, 2022
@icezyclon
Copy link
Author

As a follow up question I wanted to ask:
Is there a way to make virtualenv install into myenv/bin instead of myenv/local/binagain, possibly ignoring the posix_local prefix?
Thank you! 💯

@d3al
Copy link

d3al commented May 7, 2022

Can confirm, I'm having the same issue with PopOS 22.04

@gaborbernat
Copy link
Contributor

Posting a docker image that reproduces this would be helpful here.

@icezyclon
Copy link
Author

Posting a docker image that reproduces this would be helpful here.

I am not an expert with Docker, but I could not reproduce this bug in the ubuntu:22.04 image.
This kind of does mean that it is not just sysconfig.get_default_scheme() == 'posix_local' that is responsible for the bug, as virtualenv does install into myenv/bin in the Docker image...

However, the Ubuntu Docker image is much more slim and includes way fewer debs than the Ubuntu Desktop ISO and of course no Gnome or similar. There could easily be some configuration in the Desktop ISO that is not included in the Docker image.

What I tried:

FROM ubuntu:22.04

RUN	apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    python3 \
    python3-pip \
    python3-venv \
    python3-virtualenv \
    && apt-get clean

CMD ["/usr/bin/bash"]

putting the above into a Dockerfile in a directy, then building and running it with:

docker build --tag=virtualenvtest .
docker run -it virtualenvtest
python3 -c "import sysconfig; print(sysconfig.get_default_scheme())"
virtualenv --version
virtualenv --creator=venv myenv -vvv --with-traceback
ls myenv/bin
ls myenv/local/bin   # says there is no myenv/local directory

Again, this does not reproduce the bug.
If someone else could take a look that would be great.

We do have someone with the same problem here so it does seem to not only be on my system.

@bozhodimitrov
Copy link

Similar issue manifested for me as well, after manual upgrade from 21.04 to 22.04, so it might be related to the upgrade process.
If it is not reproducible on a clean 22.04 container, maybe try to upgrade existing container from 21.04 to 22.04.

For me, the result is that I get a complete random list of files/dirs inside my virtualenv dir.

Sometimes it's like this:

user@host:~$ ls -a venv
.  ..  .gitignore  lib  local  pyvenv.cfg

Sometimes, there is a bin folder with just the symlinks ...

user@host:~$ ls venv/bin/
python  python3  python3.10

I am using the latest available virtualenv from https://bootstrap.pypa.io/virtualenv.pyz

@icezyclon
Copy link
Author

Similar issue manifested for me as well, after manual upgrade from 21.04 to 22.04, so it might be related to the upgrade process. [...]

I just wanted to add that my 22.04 installation was a fresh install, so it might not be directly related to the upgrade process but some settings or config from 22.04 in general.

@d3al
Copy link

d3al commented May 8, 2022

I wanted to add. I initially upgraded from PopOS 21.10 to 22.04 and that's when things fell apart. My django project wouldn't run at all, couldn't update pipenv packages so I recreated my interpreter and it failed. I had other issues since the upgrade not related to virtualenv so I reinstall the OS.

I did a refresh, so the OS was reinstalled, which keeps your home directory but I lost all of my apps. Issue with virtual env was there but the other issues and errors went away. I'm getting the /bin and /lib folders inside a local folder when I run virtualenv manually. My PyCharm IDE fails to create the env as well. I can install the OS on a virtual machine and see if I get the same results.

@d3al
Copy link

d3al commented May 9, 2022

I installed PopOS 22.04 on a virtual machine and did two tests, the first was with no system updates after install and the other was after installing system updates. Both test created the lib and bin in .venv/.

sudo apt-get install python3-pip
sudo pip3 install virtualenv
mkdir project
cd project
virtualenv .venv

On my machine, I ran the same commands and I get a lib and local folder. Inside the local I have lib and bin.
I also ran the commands again as sudo and got the local folder again.

I'm not sure what I can do to help.

@icezyclon
Copy link
Author

I believe that if we knew why/when virtualenv installs into the local folder, then we can get to the bottom of this.
At the very least, disabling creating this local folder should create a normal install, right?

@gaborbernat
Copy link
Contributor

PR addressing the underlying issue is welcome.

@thenewnano
Copy link

if we are to attempt a fix, we need a bit of guidance as to where should we start?

@gaborbernat
Copy link
Contributor

Finding out what is happening and how to fix it is like 80% of the work needed here 😅 so what questions do you have?

@thenewnano
Copy link

Finding out what is happening and how to fix it is like 80% of the work needed here 😅 so what questions do you have?

Obviously it is! I would like to know where the folder path is decided in the code, where the local/bin is created.

@gaborbernat
Copy link
Contributor

gaborbernat commented Jul 18, 2022

Will be in the creator code here somewhere https://github.com/pypa/virtualenv/blob/main/src/virtualenv/create/via_global_ref/builtin/via_global_self_do.py that likely uses the information feeded by https://github.com/pypa/virtualenv/blob/main/src/virtualenv/discovery/py_info.py. Per the logs above CPython3Posix is the creator, so will be somewhere in its inheritance chain.

@d3al
Copy link

d3al commented Jul 18, 2022 via email

@rkucsora
Copy link
Contributor

I could create a Dockerfile to reproduce the issue. It requires setuptools>=60.0.0

FROM ubuntu:22.04

RUN apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    python3.10 \
    python3-pip \
    python3-virtualenv

RUN pip install setuptools==60.0.0
RUN virtualenv --creator=venv myenv 
CMD ["ls", "-a", "myenv"]

It prints

bin  include  lib  lib64  local  pyvenv.cfg

There are both bin and local directories. Checking python and pip as mentioned in the first post:

root@319a6ef851db:/# source myenv/bin/activate
(myenv) root@319a6ef851db:/# which python3 && which pip3
/myenv/bin/python3
/usr/bin/pip3
(myenv) root@319a6ef851db:/# deactivate
root@319a6ef851db:/# source myenv/local/bin/activate
(myenv) root@319a6ef851db:/# which python3 && which pip3
/usr/bin/python3
/myenv/local/bin/pip3

@gaborbernat
Copy link
Contributor

@rkucsora do you see the same behaviour if you use the builtin creator instead of the venv one?

@rkucsora
Copy link
Contributor

Not the same, in that case there is only a local directory.

root@cd7a43210371:~# virtualenv venv
created virtual environment CPython3.10.4.final.0-64 in 527ms
  creator CPython3Posix(dest=/root/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
root@cd7a43210371:~# ls venv
lib  local  pyvenv.cfg
root@cd7a43210371:~# source venv/local/bin/activate
(venv) root@cd7a43210371:~# which python3 && which pip3
/root/venv/local/bin/python3
/root/venv/local/bin/pip3
(venv) root@cd7a43210371:~# which python && which pip
/root/venv/local/bin/python
/root/venv/local/bin/pip

@gaborbernat
Copy link
Contributor

This project is only responsible of the builtin creator. The venv creator is managed by debian so for that you're looking at filling a bug for that projects tracker.

jgarver added a commit to NVIDIA/edk2-nvidia that referenced this issue Sep 6, 2024
This works-around this virtualenv bug:
pypa/virtualenv#2340

This issue came to our attention via:
#104

Signed-off-by: Jake Garver <[email protected]>
Reviewed-by: Jeff Brasen <[email protected]>
Reviewed-by: Ashish Singhal <[email protected]>
Reviewed-by: Jeshua Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants