pip fails on python 3.10 + windows

[jieru-hu]

Current Behavior

(hydra) PS C:\Users\circleci\project> nox -s lint-3.10
nox -s lint-3.10
nox > Running session lint-3.10
nox > Creating virtual environment (virtualenv) using python.EXE in .nox\lint-3-10
Operating system	:	Windows
NOX_PYTHON_VERSIONS	:	['3.10']
nox > python -m pip install --upgrade pip
nox > Command python -m pip install --upgrade pip failed with exit code 1:
Traceback (most recent call last):
  File "C:\tools\miniconda3\envs\hydra\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\tools\miniconda3\envs\hydra\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\cli\main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\cli\cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\cli\parser.py", line 12, in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\configuration.py", line 24, in <module>
    from pip._internal.utils import appdirs
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_internal\utils\appdirs.py", line 13, in <module>
    from pip._vendor import platformdirs as _appdirs
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 33, in <module>
    PlatformDirs = _set_platform_dir_class()  #: Currently active platform
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 29, in _set_platform_dir_class
    result: type[PlatformDirsABC] = getattr(importlib.import_module(module), name)
  File "C:\tools\miniconda3\envs\hydra\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\Users\circleci\project\.nox\lint-3-10\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 3, in <module>
    import ctypes
  File "C:\tools\miniconda3\envs\hydra\lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

I can run pip on the same machine with no issue outside of nox

(hydra) PS C:\Users\circleci> python --version
python --version
Python 3.10.4
(hydra) PS C:\Users\circleci\project> python -m pip install --upgrade pip
python -m pip install --upgrade pip
Requirement already satisfied: pip in c:\tools\miniconda3\envs\hydra\lib\site-packages (22.1)

on the same machines, other python versions works as expected.

(hydra) PS C:\Users\circleci\project> nox -s lint-3.9 
nox -s lint-3.9
nox > Running session lint-3.9
nox > Creating virtual environment (virtualenv) using python.exe in .nox\lint-3-9
nox > python -m pip install --upgrade pip
nox > python -m pip install --upgrade setuptools
...

Expected Behavior

nox should work on py 3.10 + windows.

Steps To Reproduce

here is an example failures job we have on our CI. https://app.circleci.com/pipelines/github/facebookresearch/hydra/14406/workflows/26c69463-cd46-40ad-a8f1-771e0bfd33a9/jobs/140853

You should be able to repro it by submitting a PR in our repo.

Environment

- OS: windows
- Python: 3.10.4
- Nox: 2022.1.7


more system info in case it helps:

(hydra) PS C:\Users\circleci\project> systeminfo
systeminfo

Host Name:                 PACKER-623CA52B
OS Name:                   Microsoft Windows Server 2019 Datacenter
OS Version:                10.0.17763 N/A Build 17763
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Standalone Server
OS Build Type:             Multiprocessor Free
Registered Owner:          N/A
Registered Organization:   N/A
Product ID:                00430-00000-00000-AA835
Original Install Date:     3/24/2022, 5:09:54 PM
System Boot Time:          5/12/2022, 10:45:21 PM
System Manufacturer:       Google
System Model:              Google Compute Engine
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 63 Stepping 0 GenuineIntel ~2300 Mhz
BIOS Version:              Google Google, 1/1/2011
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume2
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC+00:00) Monrovia, Reykjavik
Total Physical Memory:     15,356 MB
Available Physical Memory: 10,910 MB
Virtual Memory: Max Size:  16,380 MB
Virtual Memory: Available: 12,012 MB
Virtual Memory: In Use:    4,368 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    WORKGROUP
Logon Server:              \\PACKER-623CA52B
Hotfix(s):                 10 Hotfix(s) Installed.
                           [01]: KB5010473
                           [02]: KB4470788
                           [03]: KB4486153
                           [04]: KB4524244
                           [05]: KB4535680
                           [06]: KB4562562
                           [07]: KB4577586
                           [08]: KB4589208
                           [09]: KB5011503
                           [10]: KB5009642
Network Card(s):           2 NIC(s) Installed.
                           [01]: Google VirtIO Ethernet Adapter
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     169.254.169.254
                                 IP address(es)
                                 [01]: 10.142.0.130
                                 [02]: fe80::9a5:52ac:99fa:df30
                           [02]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (nat)
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 172.27.192.1
                                 [02]: fe80::8c2:4b00:254b:a7bf
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Anything else?

any pointer would be awesome... thanks for your amazing work!!

[DiddiLeija]

I don't really think this is a Nox issue. Looking at the output, the error didn't came from a Nox operation, so I assume it came from somewhere else. Does a pip install fails on a Python 3.10 virtual environment ¹?

Footnotes

1. Just to be clear, try a package that is not installed (using pip itself is not bad, but you'll might get a Requirement already satisfied message instead of a real installation process). Also, I see that Nox made your virtual environments using virtualenv, so you might have to use this tool, just to reproduce the situation as far as possible :) ↩

[jieru-hu]

hi @DiddiLeija thanks for the suggestion. pip also succeeds in virtualenv as well which is why i suspect this might be nox related.

(hydra) PS C:\Users\circleci> python -m venv venv
python -m venv venv
(hydra) PS C:\Users\circleci> which python
/c/tools/miniconda3/envs/hydra/python
(hydra) PS C:\Users\circleci> python --version
python --version
Python 3.10.4
(hydra) PS C:\Users\circleci> venv\Scripts\Activate.ps1
venv\Scripts\Activate.ps1
(venv) (hydra) PS C:\Users\circleci> which python
which python
/c/Users/circleci/venv/Scripts/python
(venv) (hydra) PS C:\Users\circleci> python -m pip install --upgrade pip
python -m pip install --upgrade pip
Requirement already satisfied: pip in c:\users\circleci\venv\lib\site-packages (22.0.4)
Collecting pip
  Using cached pip-22.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.1

[DiddiLeija]

Well, that's weird. This is what I suppose: the issue comes from a somehow broken venv, the question is where.

Your Python is managed via conda (miniconda, actually), but you create the venv via python -m virtualenv. Just to discard a possibility, can you set up Nox to create the venv with conda?

[jieru-hu]

thanks @DiddiLeija for the suggestion! I will give it a try. TIL could use conda as the backend :P

[FollowTheProcess]

@jieru-hu this looks like some conda weirdness to me, IIRC conda does a lot of complex stuff with compiled stuff and linked libraries in the environment such that it "just works" which it does almost all the time in fairness, just means when it does go wrong it happens in a weird way. I've had trouble in the past mixing conda and virtualenv in the same environment.

I'd suggest using conda as the venv backend and report back. Unfortunately I don't have a windows machine to debug on so you actually may be better placed than me to experiment!

[DiddiLeija]

Yeah, I also think this came from mixing conda and virtualenv. So that's why we have venv_backend, more info here.

[DiddiLeija]

Hi @jieru-hu! It's been a while since my last comment here, so did you manage to fix your issue? Please let us know 🙃