Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Errors when running build from tox installed in a Conda env on Windows 10 #253

Closed
abravalheri opened this issue Mar 9, 2021 · 7 comments
Closed

Errors when running build from tox installed in a Conda env on Windows 10 #253

abravalheri opened this issue Mar 9, 2021 · 7 comments

Comments

@abravalheri
Copy link
Collaborator

@pyscaffold, we have a tox.ini template that uses build to do the packaging, but recently we have identified in a CI run that it fails in the combination of Windows + Conda + tox (i.e. when we use miniconda to obtain a specific python version on Windows, and then use tox to run build).

Do you guys have any ideas or tips on how to fix or workaround this behaviour?

The steps to reproduce the error on Windows' PowerShell are:

> cd $env:LOCALAPPDATA\Temp
> conda create -p .\.conda python=3.8
> conda run -p .\.conda\ conda install tox -c conda-forge
> git clone https://github.com/pyscaffold/pyscaffold-demo.git
> cd .\pyscaffold-demo\
> conda run -p ..\.conda\ tox -e build

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['tox', '-e', 'build']' command failed.  (See above for error)
build create: $env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build
build installdeps: build
build installed: build==0.3.0,packaging==20.9,pep517==0.9.1,pyparsing==2.4.7,toml==0.10.2
build run-test-pre: PYTHONHASHSEED='353'
build run-test: commands[0] | python -m build .
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvlauncher.EXE'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvlauncher.exe'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvwlauncher.exe'
Traceback (most recent call last):
  File "$env:LOCALAPPDATA\Temp\.conda\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\venv\__init__.py", line 68, in create
    self._setup_pip(context)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\venv\__init__.py", line 294, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 1311, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
ERROR: InvocationError for command '$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\Scripts\python.EXE' -m build . (exited with code 1)
___________________________________ summary ___________________________________
ERROR:   build: commands failed

If we later just tap into the virtualenv created by tox and try to run build directly we end up seeing a very similar error:

> .\.tox\build\Scripts\python.exe -m build .

Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvlauncher.exe'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvlauncher.exe'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.tox\\build\\Scripts\\venvwlauncher.exe'
Traceback (most recent call last):
  File "$env:LOCALAPPDATA\Temp\.conda\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.tox\build\lib\site-packages\build\env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\venv\__init__.py", line 68, in create
    self._setup_pip(context)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\venv\__init__.py", line 294, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "$env:LOCALAPPDATA\Temp\.conda\lib\subprocess.py", line 1311, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

The rationale behind using this chain of tools are:

  • conda: allows users to install multiple (and coexisting) versions of Python on Windows, which is very handy when multiple projects requires the developers to stick with a specific version
  • tox: as an automation tool, it allows devs to simplify workflows... As long as they have tox installed, they don't have to worry about dependencies for specific tasks (e.g. having to install build when you want to build, pep8 / mypy when you want to lint, sphinx when you want to build the docs and etc...)

which I suppose is not too much of a stretch of a Python's developer workflow (specially considering data science devs working on Windows).

Since tox uses virtualenv under the hood, I also tried to repeat the process with virtualenv directly, finding the same error:

> conda run -p ..\.conda\ conda install virtualenv -c conda-forge
> conda run -p ..\.conda\ virtualenv .venv
> .\.venv\Scripts\python.exe -m pip install build
> .\.venv\Scripts\python.exe -m build .

Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.venv\\Scripts\\venvlauncher.exe'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.venv\\Scripts\\venvlauncher.exe'
Unable to copy '$env:LOCALAPPDATA\\Temp\\pyscaffold-demo\\.venv\\Scripts\\venvwlauncher.exe'
Traceback (most recent call last):
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.venv\lib\site-packages\build\env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\venv\__init__.py", line 68, in create
    self._setup_pip(context)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\venv\__init__.py", line 294, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "$env:LOCALAPPDATA\Temp\pyscaffold-demo\.conda\lib\subprocess.py", line 1311, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

Additional information

Relevant tox configuration:

[testenv:{clean,build}]
description =
    Build (or clean) the package in isolation according to instructions in:
    https://setuptools.readthedocs.io/en/latest/build_meta.html#how-to-use-it
    https://github.com/pypa/pep517/issues/91
    https://github.com/pypa/build
skip_install = True
changedir = {toxinidir}
deps =
    build: build
commands =
    clean: python -c 'from shutil import rmtree; rmtree("build", True); rmtree("dist", True)'
    build: python -m build .

System information:

> conda run -p ..\.conda\ conda info

     active environment : $env:LOCALAPPDATA\Temp\.conda
    active env location : $env:LOCALAPPDATA\Temp\.conda
            shell level : 1
       user config file : $env:USERPROFILE\.condarc
 populated config files :
          conda version : 4.9.2
    conda-build version : 3.21.4
         python version : 3.8.5.final.0
       virtual packages : __win=0=0
                          __archspec=1=x86_64
       base environment : $env:USERPROFILE\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : $env:USERPROFILE\Miniconda3\pkgs
                          $env:USERPROFILE\.conda\pkgs
                          $env:LOCALAPPDATA\conda\conda\pkgs
       envs directories : $env:USERPROFILE\Miniconda3\envs
                          $env:USERPROFILE\.conda\envs
                          $env:LOCALAPPDATA\conda\conda\envs
               platform : win-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Windows/10 Windows/10.0.19041
          administrator : False
             netrc file : None
           offline mode : False

> .\.tox\build\Scripts\python.exe -V
Python 3.8.8

> .\.tox\build\Scripts\pip freeze
build==0.3.0
packaging==20.9
pep517==0.9.1
pyparsing==2.4.7
toml==0.10.2

# PowerShell version
> Get-Host | Select-Object Version

Version
-------
5.1.19041.610

# Windows Version
> [Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.19041.0 Microsoft Windows NT 10.0.19041.0
@abravalheri abravalheri mentioned this issue Mar 9, 2021
Open
@layday
Copy link
Member

layday commented Mar 9, 2021 via email

@layday
Copy link
Member

layday commented Mar 9, 2021 via email

@abravalheri
Copy link
Collaborator Author

abravalheri commented Mar 9, 2021 •
edited
Loading

I am not exactly sure if I understood your question, but yes installing and running tools inside a virtual environment created with a Python inside a conda environment seems to be working without any problems...

For example, going back to the virtual environment mentioned in the previous log, that was generated by doing:

> cd $env:LOCALAPPDATA\Temp
> conda create -p .\.conda python=3.8
...
> cd .\pyscaffold-demo\
> conda run -p ..\.conda\ conda install virtualenv -c conda-forge
> conda run -p ..\.conda\ virtualenv .venv

When I run:

> .\.venv\Scripts\pip.exe install PyScaffold django

Everything seems to install correctly:

> .\.venv\Scripts\pip.exe freeze

appdirs==1.4.4
asgiref==3.3.1 
build==0.3.0
ConfigUpdater==1.1.3
Django==3.1.7
packaging==20.9 
pep517==0.9.1
pyparsing==2.4.7
PyScaffold==4.0
pytz==2021.1
setuptools-scm==5.0.2
sqlparse==0.4.1
toml==0.10.2
tomlkit==0.7.0

And I can invoke both PyScaffold and django CLIs without no problem, not only --help but the actual project generators:

> .\.venv\Scripts\putup.exe ..\myproj
done! 🐍 🌟 ✨

> ls ..\myproj


    Directory: $env:LOCALAPPDATA\Temp\myproj


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        09/03/2021     12:31                docs
d-----        09/03/2021     12:31                src
d-----        09/03/2021     12:31                tests
-a----        09/03/2021     12:31            617 .coveragerc
-a----        09/03/2021     12:31            603 .gitignore
-a----        09/03/2021     12:31            483 .readthedocs.yml
-a----        09/03/2021     12:31             75 AUTHORS.rst
-a----        09/03/2021     12:31            138 CHANGELOG.rst
-a----        09/03/2021     12:31           1097 LICENSE.txt
-a----        09/03/2021     12:31            319 pyproject.toml
-a----        09/03/2021     12:31            315 README.rst
-a----        09/03/2021     12:31           3882 setup.cfg
-a----        09/03/2021     12:31            722 setup.py
-a----        09/03/2021     12:31           2066 tox.ini

> cd ..
> ..\pyscaffold-demo\.venv\Scripts\django-admin.exe startproject myweb .
> ls


    Directory: $env:LOCALAPPDATA\Temp\myweb


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        09/03/2021     12:33                myweb
-a----        09/03/2021     12:33            683 manage.py

@layday
Copy link
Member

layday commented Mar 9, 2021 via email

@abravalheri
Copy link
Collaborator Author

Thank you very much @layday. Everything works fine when installing build with:

> .\.venv\Scripts\pip install -I 'build[virtualenv]'

and when I edited the tox.ini file to:

[testenv:{clean,build}]
description =
    Build (or clean) the package in isolation according to instructions in:
    https://setuptools.readthedocs.io/en/latest/build_meta.html#how-to-use-it
    https://github.com/pypa/pep517/issues/91
    https://github.com/pypa/build
# NOTE: build is still experimental, please refer to the links for updates/issues
skip_install = True
changedir = {toxinidir}
deps =
-    build: build
+    build: build[virtualenv]
commands =
    clean: python -c 'from shutil import rmtree; rmtree("build", True); rmtree("dist", True)'
    build: python -m build .
# By default `build` produces wheels, you can also explicitly use the flags `--sdist` and `--wheel`

I was not aware about this extra, is there anywhere in the documentation that mentions what is the difference and gives any advice on where it is better to use one or the other?

Something that I might not have mentioned before (and might be relevant): the error just happens in that particular combination: Windows + Conda + Virtualenv.

The CI of PyScaffold and the associated feedstock on conda-forge are testing with combinations of Linux, OSX, Python directly installed, Python installed with conda. We always use tox (and therefore virtualenv, not stdlib's venv), and things seem to work well even with conda, as long as we don't touch Windows.

@abravalheri abravalheri mentioned this issue Mar 9, 2021
Merged
@uranusjr
Copy link
Member

uranusjr commented Mar 9, 2021 •
edited
Loading

This is more of an issue with Conda environments, which may not correctly create python.exe launchers in the environment. The actual behaviour is complicated and I’ve never quite figured it out, but if it does not created correctly, subsequent venv runs would fail because python.exe won’t run. You can see the same sympton showing in e.g. ContinuumIO/anaconda-issues#10822 (under other contexts), but Conda devs don’t seem to think it’s a bug but more a environment issue the end user need to troubleshoot.

Technical note: Windows API emits the same error code as FileNotFoundError (and therefore appears to raise FileNotFoundError in Python) when an executable passed to CreateProcess is not a valid executable, although technically the file is found, just not an executable as expected by CreateProcess (iow an executable is not found).

@abravalheri abravalheri mentioned this issue Mar 9, 2021
Closed
@abravalheri
Copy link
Collaborator Author

Thank you very much @uranusjr, I commented the issue in Anaconda and asked the maintainers if they prefer opening a new issue.

abravalheri added a commit to pyscaffold/pyscaffold that referenced this issue Mar 9, 2021
@abravalheri
Specify virtualenv as extra dependency for build in tox.ini
5d0b101 
`build` seems to rely by default in the stdlib's implementation of `venv`
and unfortunately for a specific combination of Windows + conda env + virtualenv (as used by tox),
that might be problematic as identified in pyscaffold/pyscaffoldext-markdown#10.

According to the discussion in pypa/build#253,
specifying `virtualenv` as an extra dependency of `build` seems to be a
good workaround. Moreover I would expect some kind speed improvements
(virtualenv is reported to be faster than venv).
@abravalheri abravalheri mentioned this issue Mar 9, 2021
Merged
abravalheri added a commit to pyscaffold/pyscaffold that referenced this issue Mar 10, 2021
@abravalheri
Specify virtualenv as extra dependency for build in tox.ini
3f3e114 
`build` seems to rely by default in the stdlib's implementation of `venv`
and unfortunately for a specific combination of Windows + conda env + virtualenv (as used by tox),
that might be problematic as identified in pyscaffold/pyscaffoldext-markdown#10.

According to the discussion in pypa/build#253,
specifying `virtualenv` as an extra dependency of `build` seems to be a
good workaround. Moreover I would expect some kind speed improvements
(virtualenv is reported to be faster than venv).
@NickleDave NickleDave mentioned this issue Nov 17, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abravalheri @uranusjr @layday