Pre-commit 2.0.0 does not work with conda based python in Windows. #1329

QuantCo-mr-T · 2020-02-11T13:41:32Z

Anaconda/miniconda based python installations require an activated conda environment before calling python or any other entry-point executable. This requirement especially affects conda running on Windows.

"pre-commit install" only stores the python executable in git hooks. This is not enough for conda based python installations if the environment of pre-commit is not activated when calling "git commit".

The following (rejected) pull request is intended to solve that problem:
#1324

QuantCo-mr-T · 2020-02-11T15:18:40Z

.pre-commit-config.yaml:

repos:
 - repo: https://github.com/Quantco/pre-commit-mirrors-black
   rev: 19.10b0
   hooks:
     - id: black-conda
       args:
         - --safe
         - --target-version=py36

Good case with running git in activated conda environment 'pre-commit':

(pre-commit) C:\Temp\code\try_pre_commit>D:\Programs\Miniconda3\condabin\conda activate pre-commit

(pre-commit) C:\Temp\code\try_pre_commit>pre-commit install
pre-commit installed at .git\hooks\pre-commit

(pre-commit) C:\Temp\code\try_pre_commit>"C:\Program Files\Git\bin\git" commit
black-conda..........................................(no files to check)Skipped
On branch master
Untracked files:
        t.sh

nothing added to commit but untracked files present

(pre-commit) C:\Temp\code\try_pre_commit>

Bad case when deactivating environment right afterwards:

(pre-commit) C:\Temp\code\try_pre_commit>D:\Programs\Miniconda3\condabin\conda deactivate

C:\Temp\code\try_pre_commit>"C:\Program Files\Git\bin\git" commit
Traceback (most recent call last):
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\__main__.py", line 1, in <module>
    from pre_commit.main import main
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\main.py", line 13, in <module>
    from pre_commit.commands.autoupdate import autoupdate
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\commands\autoupdate.py", line 17, in <module>
    from pre_commit.clientlib import InvalidManifestError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\clientlib.py", line 16, in <module>
    from pre_commit.error_handler import FatalError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\error_handler.py", line 10, in <module>
    from pre_commit.store import Store
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\store.py", line 4, in <module>
    import sqlite3
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.

asottile · 2020-02-11T15:20:26Z

this is a bug with conda, I'd suggest opening an issue there

QuantCo-mr-T · 2020-02-11T15:20:46Z

The problem of running git within activated python environment where pre-commit is installed arises when using an IDE like PyCharm. There it is easy to configure the activation of different conda environments when running python, but it is not considered a useful feature for calling git

QuantCo-mr-T · 2020-02-11T15:22:26Z

this is a bug with conda, I'd suggest opening an issue there

This is expected behavior for conda. You are not supposed to call a python installed by conda without activating that environment. I hate that behavior, but I like to use the powerful package manager capabilities which span beyond python dependencies..

QuantCo-mr-T · 2020-02-11T15:23:20Z

pre-commit only extracts the path to the python executable when "pre-commit install" is called. This is not enough for a conda installed python.

asottile · 2020-02-11T15:24:07Z

then conda is violating some basic expectations of how python works and should be fixed

asottile · 2020-02-11T15:25:24Z

A workaround may be for you to suggest your patch to the conda packaging of pre-commit, but I do not want to support it here

QuantCo-mr-T · 2020-02-11T15:25:41Z

I agree. I have ideas how conda should work instead. But I see no chance of getting that through. You would need to develop an alternative with the same complexity of features.

asottile · 2020-02-11T15:26:21Z

did this work for pre-commit 1.x?

QuantCo-mr-T · 2020-02-11T15:27:25Z

I have a running system with pre-commit 1.12.0. The problem is that since more than one year, I see more and more actions with conda installed python tools to require activation.

asottile · 2020-02-11T15:28:28Z

that's not what I asked, does the problem reproduce on pre-commit 1.21.1

QuantCo-mr-T · 2020-02-11T15:28:28Z

I assume the problem is not just the pre-commit version. It may also be a change in one of pre-commit's dependencies.

asottile · 2020-02-11T15:30:05Z

I doubt it, sqlite3 has been a python standard library dependency of pre-commit since 0.4.0 (2015-02-07)

QuantCo-mr-T · 2020-02-11T15:31:17Z

It may be a version change of sqlite3. All I can say is, I have a pre-commit 1.xx working. And I have a pre-commit 1.xx installation not working. I would need to explore further.

QuantCo-mr-T · 2020-02-11T15:33:29Z

And I had one colleague where downgrading pre-commit from 2.0.0 to 1.12.0 helped.

majidaldo · 2020-07-27T20:53:03Z

this can be managed if INSTALL_PYTHON can be managed.

majidaldo · 2020-07-27T21:06:30Z

heck, i just tried a wrapped python for python -mpre_commit install and it still picked up python.exe as the INSTALL_PYTHON!

majidaldo · 2020-07-27T21:11:17Z

seriously, why does INSTALL_PYTHON have priority over which(pre-commit) ???

if os.access(INSTALL_PYTHON, os.X_OK):
    CMD = [INSTALL_PYTHON, '-mpre_commit']
elif which('pre-commit'):
    CMD = ['pre-commit']
else:
    raise SystemExit(DNE)

asottile · 2020-07-27T21:20:06Z

because it is slow to look up pre-commit on the path, and it is possible that there are other executables named pre-commit, and the python which was used to install pre-commit is the most likely to have access to pre-commit

please consider the tone of the language you're using, saying things like "seriously, why ... ???" is not appreciated

majidaldo · 2020-07-27T22:22:04Z

because it is slow to look up pre-commit on the path, and it is possible that there are other executables named pre-commit, and the python which was used to install pre-commit is the most likely to have access to pre-commit

please consider the tone of the language you're using, saying things like "seriously, why ... ???" is not appreciated

apologies.

majidaldo · 2020-09-30T00:03:26Z

fwiw, you can do

if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`

...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.

asottile · 2020-10-12T15:10:49Z

shell=True is unsafe, you've introduced a security problem into your script. please don't suggest insecure solutions

kai-tub · 2020-11-21T10:31:18Z

Thanks for the fix @gabrielecalvo 🎉
For anyone else who gets the following error

An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\...\\Miniconda3\\envs\\ENV\\python.exe', '-mvirtualenv', 'C:\\Users\\...\\.cache\\pre-commit\\repo4xuok9ga\\py_env-python3.8')
return code: 1
expected return code: 0
stdout:
    FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\...\\Miniconda3\\envs\\ENV\\Lib\\venv\\scripts\\nt\\python.exe'

See the virtualenv in conda issue #10822.

sarthakpati · 2021-05-24T13:25:54Z

fwiw, you can do
if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`
...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.

Hi @majidaldo, do you have a solution that uses exec-wrappers?

majidaldo · 2021-05-24T14:01:43Z

fwiw, you can do
if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`
...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.
Hi @majidaldo, do you have a solution that uses exec-wrappers?

yes. essentially replace INSTALL_PYTHON in the git hook script with the wrapper.

sarthakpati · 2021-05-24T14:25:11Z

fwiw, you can do
if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`
...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.
Hi @majidaldo, do you have a solution that uses exec-wrappers?
yes. essentially replace INSTALL_PYTHON in the git hook script with the wrapper.

Awesome! Do you have an example or a repo you can point me to?

majidaldo · 2021-05-24T18:53:55Z

fwiw, you can do
if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`
...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.
Hi @majidaldo, do you have a solution that uses exec-wrappers?
yes. essentially replace INSTALL_PYTHON in the git hook script with the wrapper.
Awesome! Do you have an example or a repo you can point me to?

unfortunately, it's not public (yet).

side note: since you're a data sciency user like i am, you can manage a fairly sophisticated data science project set up by using multiple conda environments simultaneously. but i only wanted 1 git and 1 corresponding precommit install; that's why i was interested in this. it's the solution that i'd like to make public.

sarthakpati · 2021-05-24T18:59:15Z

fwiw, you can do
if which('pre-commit') and which('conda'):
    CMD = [which('conda'), 'run', '-n', os.environ.get('CONDA_DEFAULT_ENV'), 'pre-commit']`
...but it's very slow and the output isn't nicely formatted. otherwise, use conda wrappers from exec-wrappers.
Hi @majidaldo, do you have a solution that uses exec-wrappers?
yes. essentially replace INSTALL_PYTHON in the git hook script with the wrapper.
Awesome! Do you have an example or a repo you can point me to?
unfortunately, it's not public (yet).

side note: since you're a data sciency user like i am, you can manage a fairly sophisticated data science project set up by using multiple conda environments simultaneously. but i only wanted 1 git and 1 corresponding precommit install; that's why i was interested in this. it's the solution that i'd like to make public.

I am definitely interested in that solution. Anything else is going to be liable to break far too easily. If you are interested, we can collaborate on this on the project I am interested in?

sarthakpati · 2021-06-10T14:12:10Z

Hey @majidaldo have you made the solution public?

majidaldo · 2021-06-11T19:14:11Z

Hey @majidaldo have you made the solution public?

hey let's take this conversation in private messages so we don't offtopic this thread.

henhuy · 2021-06-14T13:48:05Z

For me, adding following line after initialization of INSTALL_PYTHON in pre-commit file worked:
os.environ['PATH'] = f'{os.path.dirname(INSTALL_PYTHON)}{os.pathsep}{os.environ["PATH"]}'
I found this solution somewhere, but unfortunately I cannot find link anymore.
Don't know how stable this solution is - but seems easy, and safe - so wanted to post it. Maybe this helps someone...

xhochy · 2021-12-30T07:14:09Z

This has been fixed in conda-forge for Python 3.9+

JaGarRod · 2022-01-10T12:30:33Z

This has been fixed in conda-forge for Python 3.9+

Could you explain what exactly has been fixed or point to the changes you refer to?

xhochy · 2022-01-10T12:32:14Z

You can now run Python again without the need to activate the conda environment. You can see that changes at conda-forge/python-feedstock#532 but they are quite extensive.

ChristianZimpelmann · 2022-01-18T06:21:42Z

I still got the same error using a python 3.9.9 environment (see below). Strangely, it worked when I used another conda environment based on python 3.8.

[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('C:\\Users\\Christian Zimpelmann\\.cache\\pre-commit\\repock9m01yq\\py_env-default\\Scripts\\python.EXE', '-mpip', 'install', '.')
return code: 1
expected return code: 0
stdout: (none)
stderr:
    Traceback (most recent call last):
      File "C:\.conda\envs\ambig-beliefs\lib\runpy.py", line 197, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "C:\.conda\envs\ambig-beliefs\lib\runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\__main__.py", line 29, in <module>
        from pip._internal.cli.main import main as _main
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\cli\main.py", line 9, in <module>
        from pip._internal.cli.autocompletion import autocomplete
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\cli\autocompletion.py", line 10, in <module>
        from pip._internal.cli.main_parser import create_main_parser
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\cli\main_parser.py", line 8, in <module>
        from pip._internal.cli import cmdoptions
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\cli\cmdoptions.py", line 23, in <module>
        from pip._internal.cli.parser import ConfigOptionParser
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\cli\parser.py", line 12, in <module>
        from pip._internal.configuration import Configuration, ConfigurationError
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\configuration.py", line 20, in <module>
        from pip._internal.exceptions import (
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_internal\exceptions.py", line 7, in <module>
        from pip._vendor.pkg_resources import Distribution
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 80, in <module>
        from pip._vendor import platformdirs
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 31, in <module>
        PlatformDirs = _set_platform_dir_class()  #: Currently active platform
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_vendor\platformdirs\__init__.py", line 27, in _set_platform_dir_class
        result: Type[PlatformDirsABC] = getattr(importlib.import_module(module), name)
      File "C:\.conda\envs\ambig-beliefs\lib\importlib\__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "C:\Users\Christian Zimpelmann\.cache\pre-commit\repock9m01yq\py_env-default\lib\site-packages\pip\_vendor\platformdirs\windows.py", line 1, in <module>
        import ctypes
      File "C:\.conda\envs\ambig-beliefs\lib\ctypes\__init__.py", line 8, in <module>
        from _ctypes import Union, Structure, Array
    ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

xhochy · 2022-01-18T06:59:37Z

@ChristianZimpelmann This doesn't work with any Python 3.9.9 but only the latest build from conda-forge. Can you share your conda list -p C:\.conda\envs\ambig-beliefs output as a new issue in https://github.com/conda-forge/python-feedstock ?

asottile closed this as completed Feb 11, 2020

asottile added upstream-bug windows labels Feb 11, 2020

QuantCo-mr-T mentioned this issue Feb 14, 2020

Conda installed pre-commit causes problems with graphical git front-ends conda-forge/pre-commit-feedstock#9

Closed

This comment has been minimized.

Sign in to view

lorenzwalthert mentioned this issue Jan 30, 2021

Getting "permission denied" when trying to commit after setting up precommit hooks lorenzwalthert/precommit#221

Closed

asottile mentioned this issue May 23, 2021

Conda installation with Windows not working #1920

Closed

sarthakpati mentioned this issue May 24, 2021

Source formatting standardization mlcommons/GaNDLF#52

Closed

jpn-- mentioned this issue Nov 19, 2021

Black and iSort Code Formatting ActivitySim/activitysim#488

Closed

ChristianZimpelmann mentioned this issue Jan 18, 2022

conda under Windows: DLL load failed when running pre-commit hooks conda-forge/python-feedstock#539

Closed

metab0t mentioned this issue Jan 30, 2022

PyCall cannot find ctypes on python 3.10.2 JuliaPy/PyCall.jl#963

Closed

qci-amos mentioned this issue May 3, 2022

using pre-commit install from WSL #2371

Closed

csadorf mentioned this issue May 30, 2022

Errors when running pre-commit hooks materials-marketplace/standard-app-api#6

Closed

tomaarsen mentioned this issue Jul 27, 2022

Pre commit hook fails nltk/nltk#3027

Closed

Pre-commit 2.0.0 does not work with conda based python in Windows. #1329

Pre-commit 2.0.0 does not work with conda based python in Windows. #1329

Comments

QuantCo-mr-T commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

asottile commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

asottile commented Feb 11, 2020

asottile commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020 • edited Loading

asottile commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

asottile commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

asottile commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

QuantCo-mr-T commented Feb 11, 2020

majidaldo commented Jul 27, 2020

majidaldo commented Jul 27, 2020 • edited Loading

majidaldo commented Jul 27, 2020

asottile commented Jul 27, 2020

majidaldo commented Jul 27, 2020

majidaldo commented Sep 30, 2020 • edited Loading

This comment has been minimized.

asottile commented Oct 12, 2020

kai-tub commented Nov 21, 2020

sarthakpati commented May 24, 2021

majidaldo commented May 24, 2021 • edited Loading

sarthakpati commented May 24, 2021 • edited Loading

majidaldo commented May 24, 2021

sarthakpati commented May 24, 2021

sarthakpati commented Jun 10, 2021

majidaldo commented Jun 11, 2021

henhuy commented Jun 14, 2021

xhochy commented Dec 30, 2021

JaGarRod commented Jan 10, 2022

xhochy commented Jan 10, 2022

ChristianZimpelmann commented Jan 18, 2022 • edited Loading

xhochy commented Jan 18, 2022

QuantCo-mr-T commented Feb 11, 2020 •

edited

Loading

majidaldo commented Jul 27, 2020 •

edited

Loading

majidaldo commented Sep 30, 2020 •

edited

Loading

majidaldo commented May 24, 2021 •

edited

Loading

sarthakpati commented May 24, 2021 •

edited

Loading

ChristianZimpelmann commented Jan 18, 2022 •

edited

Loading