Add pyenv-win support

[Cellorator]

I tried making a virtual environment with:
pipenv --python 3.8.6

It asked me this:

Warning: Python 3.8.6 was not found on your system...
Would you like us to install CPython 3.8.6 with Pyenv? [Y/n]:

in which I said yes, but it threw me these errors:

Traceback (most recent call last):
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Scripts\pipenv.exe\__main__.py", line 7, in <module>
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 1236, in invoke
    return Command.invoke(self, ctx)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Carl\.pyenv\pyenv-win\versions\3.9.0\Lib\site-packages\pipenv\vendor\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\cli\command.py", line 198, in cli
    ensure_project(
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\core.py", line 576, in ensure_project
    ensure_virtualenv(
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\core.py", line 498, in ensure_virtualenv
    python = ensure_python(three=three, python=python)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\core.py", line 451, in ensure_python
    c = installer.install(version)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\installers.py", line 191, in install
    c = self._run(
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\installers.py", line 124, in _run
    c = delegator.run(args, block=False, timeout=timeout)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\vendor\delegator.py", line 336, in run
    c.run(block=block, binary=binary, cwd=cwd, env=env)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\vendor\delegator.py", line 204, in run
    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\site-packages\pipenv\vendor\pexpect\popen_spawn.py", line 53, in __init__
    self.proc = subprocess.Popen(cmd, **kwargs)
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "c:\users\carl\.pyenv\pyenv-win\versions\3.9.0\lib\subprocess.py", line 1416, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 is not a valid Win32 application

Pipenv Version: 2020.11.4
Pyenv-Win Version: 2.64.3
Python Version: 3.9.0

[Mause]

Is pyenv-win on your path?

[Cellorator]

@Mause
Yes

[uranusjr]

pyenv-win is not supported; the main issue being it using a BAT script as a command, and is only executable directly in the command prompt, not as a separate process (i.e. the CreateProcess API). It is definitely possible to add support to it (use subprocess.Popen(..., shell=True) when pipenv.installers.Installer._find_python_installer_by_name_and_env returns a .bat file on Windows, but someone needs to write the code.

[Cellorator]

@uranusjr All right thx. I'll just install the correct python version before making the virtual environment for now

[MrShiny608]

Hey, I've been having similar issues.

Mine seem to be related to how pipenv detects currently installed Python versions, I am using pyenv-win as well, but I have already installed the version required. From looking into the code it looks like pipenv expects a registry entry for each installed Python version which pyenv has not created. Do you think this is a pyenv issue? Or should pipenv be looking at pyenv versions as it has already found pyenv is in use?

pyenv --version
pyenv 2.64.3

pipenv --version
pipenv, version 2020.11.15

pyenv global
3.7.7

pyenv versions
* 3.7.7 (set by C:\Users\tim.brookes\.pyenv\pyenv-win\version)
  3.8.2

With Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
asyncio = "==3.4.3"

[requires]
python_full_version = "3.8.2"

Running pipenv --rm || true && pipenv install causes the following output

pipenv --rm || true && pipenv install
Removing virtualenv (C:\Users\tim.brookes\.virtualenvs\application_server-Lr3JGuqK)...
[=   ] Running..Warning: Python 3.8.2 was not found on your system...
Creating a virtualenv for this project...
Pipfile: C:\work\workspace\admin-panel-combined\application_server\Pipfile
Using default python from c:\users\tim.brookes\.pyenv\pyenv-win\versions\3.7.7\python.exe (3.7.7) to create virtualenv...
[  ==] Creating virtual environment...created virtual environment CPython3.7.7.final.0-64 in 523ms
  creator CPython3Windows(dest=C:\Users\tim.brookes\.virtualenvs\application_server-Lr3JGuqK, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\tim.brookes\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Successfully created virtual environment!
Virtualenv location: C:\Users\tim.brookes\.virtualenvs\application_server-Lr3JGuqK
Warning: Your Pipfile requires python_version 3.8.2, but you are using 3.7.7 (C:\Users\t\.\a\S\python.exe).
  $ pipenv --rm and rebuilding the virtual environment may resolve the issue.
  $ pipenv check will surely fail.
Installing dependencies from Pipfile.lock (a4028d)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

[AnnanFay]

@MrShiny608 I think you need to set the local python version to 3.8.2 using pyenv before running pipenv.

So you want pyenv global=3.7.7, local=3.8.2, then run pipenv which will now use the local 3.8.2.

From what I can tell Pipenv with pyenv-win doesn't have any knowledge of what is installed and will only see the currently active python. So with 3.7.7 activated Pipenv thinks you only have that installed. Though the documentation is pretty lacking.

My output from pipenv install is:

Creating a virtualenv for this project...
Pipfile: F:\projects\project-name\Pipfile
Using c:/users/annan/.pyenv/pyenv-win/shims/python.bat (3.8.6) to create virtualenv...
[  ==] Creating virtual environment...created virtual environment CPython3.8.6.final.0-64 in 6772ms
  creator CPython3Windows(dest=C:\Users\annan\.virtualenvs\project-name-KqztLqDk, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\annan\AppData\Local\pypa\virtualenv)
    added seed packages: pip==20.3.1, setuptools==51.0.0, wheel==0.36.1
  activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Successfully created virtual environment!
Virtualenv location: C:\Users\annan\.virtualenvs\project-name-KqztLqDk
Installing dependencies from Pipfile.lock (dad587)...
  ================================ 6/6 - 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

The interesting bit compared to your output is it using .pyenv/pyenv-win/shims/python.bat and detecting it correctly as 3.8.6.

Also ensure that .pyenv/pyenv-win/shims is early in your $PATH.

[i3v]

I just ran into this too, and I think I was able to make some minor progress with it. Just to document this:

1. I get:

  File "c:\users\username\appdata\local\programs\python\python36\lib\site-packages\pipenv\installers.py", line 124, in _run
    c = delegator.run(args, block=False, timeout=timeout)
  File "c:\users\username\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\delegator.py", line 336, in run
    c.run(block=block, binary=binary, cwd=cwd, env=env)
  File "c:\users\username\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\delegator.py", line 204, in run
    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
  File "c:\users\username\appdata\local\programs\python\python36\lib\site-packages\pipenv\vendor\pexpect\popen_spawn.py", line 53, in __init__
    self.proc = subprocess.Popen(cmd, **kwargs)
  File "c:\users\username\appdata\local\programs\python\python36\lib\subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "c:\users\username\appdata\local\programs\python\python36\lib\subprocess.py", line 1017, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

2. Quick googling suggests to replace subprocess.Popen(cmd, **kwargs) with subprocess.Popen(cmd, **kwargs, shell=True) here. Indeed, this allowed me to make a step further. (And, IMHO, that could be a fair compatibility fix, at least with the if os.name == 'nt': check)

3. Now I get:

W:\tests\PyTests\testenv>pipenv install
Warning: Python 3.8.2 was not found on your system...
Would you like us to install CPython 3.8.2 with Pyenv? [Y/n]: y
Installing CPython 3.8.2 with C:\Users\username\.pyenv\pyenv-win\bin\pyenv (this may take a few minutes)...
Failed...
Something went wrong...
:: [Info] ::  Mirror: https://www.python.org/ftp/python
pyenv-install: definition not found: -s

See all available versions with `pyenv install --list'.

Warning: The Python you just installed is not available on your PATH, apparently.

4. Inserting print(cmd) reveals that the first pipenv call is:
   ('C:\\Users\\username\\.pyenv\\pyenv-win\\bin\\pyenv', 'install', '--list')
   and the second is:
   ('C:\\Users\\username\\.pyenv\\pyenv-win\\bin\\pyenv', 'install', '-s', '3.8.2')

5. If I type > pyenv install -s 3.8.2 it fails with pyenv-install: definition not found: -s, just like pyenv install --skip-existing 3.8.2. The -s option seem to be a documented for at least 5 years in the original pipenv. But no effort was made from pyenv-win side to support this option.

6. If I type > pyenv install 3.8.2 it works. If I type it again, nothing bad happens - it just skips.

7. Adding Case "-s" optForce = False here seem to be enough.

8. Now I get:

W:\tests\PyTests\testenv> pipenv install
Warning: Python 3.8.2 was not found on your system...
Would you like us to install CPython 3.8.2 with Pyenv? [Y/n]: y
Installing CPython 3.8.2 with C:\Users\username\.pyenv\pyenv-win\bin\pyenv (this may take a few minutes)...
Installing python...

Success!
:: [Info] ::  Mirror: https://www.python.org/ftp/python

Warning: The Python you just installed is not available on your PATH, apparently.

This looks much like #3551 (but I've not tired to look into yet).

9. Now:

• PyCharm automatically detects the correct python environment and also correctly activates it in the built-in terminal
• pyenv versions correctly lists 3.8.2
• where pip points to C:\Users\username\.virtualenvs\testenv-pxdDyT_U\Scripts\pip.exe
• pip works (e.g. I'm able to install numpy)
• pipenv install still asks if I would like to install 3.8.2

Note:

• pyenv-win-2.64.6.1
• pipenv 2020.11.15
• base python: 3.6.8
• Windows 10 1803

[deify]

I have been trying to work on a fix for myself mainly. Sadly I don't seem to find enough time to finish it at the moment.

https://github.com/deify/pipenv/tree/pyenv-win-support

The main bugs I had were fixed and as for now I am able to use it for my use-cases. There is no sufficient test implemented yet. Also for me I had to remove a file from the existing pyenv-win install.

Maybe some1 is able to fix it properly or give me a hint on how to continue and I will have a look as soon as I find some time.

I am using the following versions:

• pyenv-win 2.64.3
• pipenv (local install from the above branch 2020.11.16.dev0 as base)
• base python 3.9.1
• Windows 10 1909

[mungojam]

pyenv-win is not supported; the main issue being it using a BAT script as a command, and is only executable directly in the command prompt, not as a separate process (i.e. the CreateProcess API). It is definitely possible to add support to it (use subprocess.Popen(..., shell=True) when pipenv.installers.Installer._find_python_installer_by_name_and_env returns a .bat file on Windows, but someone needs to write the code.

I'm no expert on this, but it looks like there are ways to use CreateProcess to run batch files, by calling cmd.exe in the CreateProcess call and passing the batch file as an argument.

See: https://stackoverflow.com/a/25919674/538403

Maybe that will make it simpler?