Conda installed pre-commit causes problems with graphical git front-ends #9

QuantCo-mr-T · 2020-02-14T21:24:16Z

Issue:
Pre-commit stores the python executable used for pre-commit install in the generated git hooks. However, a python installed by conda can only be reliably executed when the respective environment is activated (Especially when using Windows). Thus, when using pre-commit installed by this conda-forge package, the respective environment needs to be activated whenever git commit is called. However, when using a git front-end from an IDE that supports the choice between different conda installed python environments, git will not be called with the correct environment activated.

Pre-commit hook:

$ cat .git/hooks/pre-commit
#!/usr/bin/env python
# File generated by pre-commit: https://pre-commit.com
...
# start templated
INSTALL_PYTHON = 'D:\\Programs\\Miniconda3\\envs\\pre-commit\\python.exe'
...

Resulting problem reproduced in Windows cmd (activate environment pre-commit; pre-commit install; git commit -> OK; deactivate environment; git commit -> FAIL):

D:\my\code\try_pre_commit>D:\Programs\Miniconda3\condabin\activate pre-commit

D:\my\code\try_pre_commit>conda.bat activate pre-commit

(pre-commit) D:\my\code\try_pre_commit>pre-commit install
pre-commit installed at .git\hooks\pre-commit

(pre-commit) D:\my\code\try_pre_commit>git commit
black-conda..........................................(no files to check)Skipped
On branch master
nothing to commit, working tree clean

(pre-commit) D:\my\code\try_pre_commit>D:\Programs\Miniconda3\condabin\deactivate
DeprecationWarning: 'deactivate' is deprecated. Use 'conda deactivate'.

(pre-commit) D:\my\code\try_pre_commit>conda.bat deactivate

D:\my\code\try_pre_commit>git commit
Traceback (most recent call last):
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\__main__.py", line 1, in <module>
    from pre_commit.main import main
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\main.py", line 13, in <module>
    from pre_commit.commands.autoupdate import autoupdate
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\commands\autoupdate.py", line 17, in <module>
    from pre_commit.clientlib import InvalidManifestError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\clientlib.py", line 16, in <module>
    from pre_commit.error_handler import FatalError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\error_handler.py", line 10, in <module>
    from pre_commit.store import Store
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\store.py", line 4, in <module>
    import sqlite3
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.

Environment (conda list):

# packages in environment at D:\Programs\Miniconda3\envs\pre-commit:
#
# Name                    Version                   Build  Channel
aspy.yaml                 1.3.0                      py_0    conda-forge
ca-certificates           2019.11.28           hecc5488_0    conda-forge
certifi                   2019.11.28               py38_0    conda-forge
cfgv                      2.0.1                      py_0    conda-forge
editdistance              0.5.3            py38h6538335_0    conda-forge
identify                  1.4.11                     py_0    conda-forge
nodeenv                   1.3.4                      py_0    conda-forge
openssl                   1.1.1d               hfa6e2cd_0    conda-forge
pip                       20.0.2                   py38_0    conda-forge
pre-commit                2.0.0                    py38_0    conda-forge
python                    3.8.1                he1f5543_2    conda-forge
pyyaml                    5.3              py38hfa6e2cd_0    conda-forge
setuptools                45.1.0                   py38_0    conda-forge
six                       1.14.0                   py38_0    conda-forge
sqlite                    3.30.1               hfa6e2cd_0    conda-forge
toml                      0.10.0                     py_0    conda-forge
vc                        14.1                 h0510ff6_4
virtualenv                16.7.5                     py_0    conda-forge
vs2015_runtime            14.16.27012          hf0eaf9b_1
wheel                     0.34.1                   py38_0    conda-forge
wincertstore              0.2                   py38_1003    conda-forge
yaml                      0.2.2                hfa6e2cd_1    conda-forge

Details about conda and system ( conda info ):

     active environment : pre-commit
    active env location : D:\Programs\Miniconda3\envs\pre-commit
            shell level : 1
       user config file : C:\Users\me\.condarc
 populated config files :
          conda version : 4.7.12
    conda-build version : not installed
         python version : 3.7.4.final.0
       virtual packages :
       base environment : D:\Programs\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : D:\Programs\Miniconda3\pkgs
                          C:\Users\me\.conda\pkgs
                          C:\Users\me\AppData\Local\conda\conda\pkgs
       envs directories : D:\Programs\Miniconda3\envs
                          C:\Users\me\.conda\envs
                          C:\Users\me\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.4 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False

The text was updated successfully, but these errors were encountered:

QuantCo-mr-T · 2020-02-14T21:31:08Z

I tried to propose a pull request to the maintainer of pre-commit, but he considers it a bug of conda that conda installed python executables do not work out of the box:
pre-commit/pre-commit#1329

The following pull request records the conda environment used for calling pre-commit install when using a specific option and runs pre-commit hooks using conda run:
pre-commit/pre-commit#1324

It would be possible to propose a much simpler change if the conda environment is not recorded on pre-commit install, but simply taken from an environment variable that must be set when calling git commit.

nicoddemus · 2020-02-14T23:26:02Z

I tried to propose a pull request to the maintainer of pre-commit, but he considers it a bug of conda that conda installed python executables do not work out of the box:
pre-commit/pre-commit#1329

And I agree, this is mostly a conda issue, where environments need to be activated in order to be used properly. With Python virtual environments, this is never really needed, you can call python from the virtual environment and things just work.

Also, this is not something specific to pre-commit, any tool which calls a python installed by conda might suffer this.

To work around this, we use https://github.com/gqmelo/exec-wrappers (with https://github.com/conda-forge/conda-wrappers-feedstock), which basically solves the issue universally: just install the conda-wrapper package into your environment, and you will have .bat files generated for you inside your environment that activate the environment first before calling the wrapped executable. Then when you install pre-commit using the python.bat executable, it will record the full path to python.bat and things will just work.

We also use the same python.bat to register the interpreter in IDEs (such as PyCharm and LiClipse), and it works great.

cc @gqmelo

gqmelo · 2020-02-15T10:24:06Z

In general always expect anything installed using conda to work only in an activated environment.
There is a lot happening to make the packages relocatable and many of them rely on environment variables being set (and this happens during activation).

This is the reason why I created the exec-wrappers and conda-wrappers mentioned by nicoddemus. It creates wrappers that mimic the conda activation before running the real executable.

In the case of pre-commit, I think you have some options:

Use git from conda-forge + wrappers

Install git from conda-forge, create the wrappers and use the git wrapper instead of the git installed in your system:

conda install git
conda install conda-wrappers

You should find a git wrapper in D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\git.bat. Running it will ensure that git runs in the activated environment.

Create wrappers for your current git installation

If you don't want to install git from conda-forge, you can wrap your current git with something like:

create-wrappers  -t conda -b C:\Program Files\Git\bin -d PATH_WHERE_TO_CREATE_WRAPPERS --conda-env-dir D:\Programs\Miniconda3\envs\pre-commit

Then use the git wrappers from PATH_WHERE_TO_CREATE_WRAPPERS

Change generated pre-commit files to use `python.bat` wrapper

Yet another option would be to change the python executable in the pre commit files to be D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.bat

This option could be easier if there were an option on pre-commit to override the python executable to be saved in the files. Currently it uses sys.executable. Maybe providing a command line option to override it is something the author is willing to accept.

QuantCo-mr-T · 2020-02-17T09:38:59Z

Thank you so much for the great hints and the well structured explanation!

The conda-wrappers package is actually pretty close to what I would see as an alternative to the conda activate system. The problem of conda activate is that it needs to hack the calling shell. I would put a pre-computed conda activate (mostly just environment variable settings) in some place of the environment on every conda install action and then have wrapper executables for all entry points that apply those before executing their real body. If they are binary injected versions of the real executables or of they include a hack changing sys.executable, conda activation would not be needed on programmatic calls.

A quick try of git.bat did not work out of the box.

D:\Programs\cygwin\home\me\code\try_pre_commit>D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\git.bat commit
Traceback (most recent call last):
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\__main__.py", line 1, in <module>
    from pre_commit.main import main
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\main.py", line 13, in <module>
    from pre_commit.commands.autoupdate import autoupdate
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\commands\autoupdate.py", line 17, in <module>
    from pre_commit.clientlib import InvalidManifestError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\clientlib.py", line 16, in <module>
    from pre_commit.error_handler import FatalError
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\error_handler.py", line 10, in <module>
    from pre_commit.store import Store
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\site-packages\pre_commit\store.py", line 4, in <module>
    import sqlite3
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "D:\Programs\Miniconda3\envs\pre-commit\lib\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.

I am missing the activate.d directory mentioned in runin.bat:

$ find /cygdrive/d/Programs/Miniconda3/envs/pre-commit -iname "*activate*"
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/common/activate
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/common/Activate.ps1
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/nt/activate.bat
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/nt/deactivate.bat
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/posix/activate.csh
/cygdrive/d/Programs/Miniconda3/envs/pre-commit/Lib/venv/scripts/posix/activate.fish

@gqmelo, did you consider using conda run? A colleague of mine tries to get a pull request upstream which would add an option to conda run that makes the output interactive:
conda/conda#9646

Calling Lib/venv/scripts/nt/activate.bat in a modified runin.bat also does not work. conda run -n pre-commit git commit does work.

lorenzwalthert · 2020-04-10T20:08:20Z

Not sure I have a similar issue (the same error message though) but when I call the precommit executable directly (full path) without activating the conda environment on Windows, it's also causing failing builds. If I get this right, stand-alone executables should not be installed with conda because they are not ultimately standalone and depend on activated envs. This sounds like bad news to me. Hard for me to investigate now, but it seems it works when there is only one conda environment because that's probably activated by default. When installing into another env, the sqlite error occurs.

lorenzwalthert · 2020-04-19T13:18:15Z

@QuantCo-mr-T I solved my problem with conda run -n env-name path/to/pre-commit/exec arg1 arg2, so I don't need to activate the environment in a separate command. This does not apply to the original issue though I think.

kylemcmearty · 2021-09-16T18:28:44Z

Then use the git wrappers from PATH_WHERE_TO_CREATE_WRAPPERS

Change generated pre-commit files to use python.bat wrapper

Yet another option would be to change the python executable in the pre commit files to be D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.bat

This option could be easier if there were an option on pre-commit to override the python executable to be saved in the files. Currently it uses sys.executable. Maybe providing a command line option to override it is something the author is willing to accept.

@gqmelo
Which file in the pre-commit package do I change to point to use the python.bat file?

Change generated pre-commit files to use python.bat wrapper

Yet another option would be to change the python executable in the pre commit files to be D:\Programs\Miniconda3\envs\pre-commit\Scripts\wrappers\conda\python.bat

kylemcmearty · 2021-09-16T19:58:47Z

In case anyone else comes across this issue, there appears to be an even simpler fix:

If using windows, activate your conda env in the terminal and then activate your editor from the terminal as well:
eg, . code for vs code or PyCharm\bin\pycharm.bat for PyCharm - suggested by @lancelote https://stackoverflow.com/questions/63663562/use-of-pre-commit-with-pycharm#comment112601701_63663562

xhochy · 2021-12-30T20:46:17Z

This should be resolved by using the latest Python 3.9 and 3.10 Python builds. No need for workarounds anymore.

master117 · 2022-10-19T14:45:22Z

This should be resolved by using the latest Python 3.9 and 3.10 Python builds. No need for workarounds anymore.

I'm facing this issue using vscode, Conda with Python 3.10.6 and pre-commit 2.20.0

xhochy · 2022-10-19T15:12:54Z

@master117 Please post a traceback or other supporting information.

master117 · 2022-10-19T15:25:41Z

@master117 Please post a traceback or other supporting information.

[2022-10-19T15:23:40.685Z] Traceback (most recent call last):
  File "C:\Users\<user>\Miniconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\<user>\Miniconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\__main__.py", line 3, in <module>
    from pre_commit.main import main
  File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\main.py", line 12, in <module>
    from pre_commit.commands.autoupdate import autoupdate
  File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\commands\autoupdate.py", line 18, in <module>
    from pre_commit.store import Store
  File "C:\Users\<user>\Miniconda3\lib\site-packages\pre_commit\store.py", line 6, in <module>
    import sqlite3
  File "C:\Users\<user>\Miniconda3\lib\sqlite3\__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "C:\Users\<user>\Miniconda3\lib\sqlite3\dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: DLL load failed while importing _sqlite3: The specified module could not be found.

If I'm stating vscode from a prompt with activate conda environemnt it works.

xhochy · 2022-10-20T14:47:41Z

Can you please also post a conda list -n base here?

master117 · 2022-10-21T08:07:33Z

Can you please also post a conda list -n base here?

Of course.

# packages in environment at C:\Users\<user>\Miniconda3:
#
# Name                    Version                   Build  Channel
brotlipy                  0.7.0           py39h2bbff1b_1003  
ca-certificates           2022.9.24            h5b45459_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.0           py39h2bbff1b_1  
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
charset-normalizer        2.0.4              pyhd3eb1b0_0  
colorama                  0.4.4              pyhd3eb1b0_0  
conda                     22.9.0           py39hcbf5309_1    conda-forge
conda-content-trust       0.1.1              pyhd3eb1b0_0  
conda-package-handling    1.8.1            py39h8cc25b3_0  
console_shortcut          0.1.1                         4  
cryptography              36.0.0           py39h21b164f_0  
distlib                   0.3.6                    pypi_0    pypi
filelock                  3.8.0              pyhd8ed1ab_0    conda-forge
identify                  2.5.6              pyhd8ed1ab_0    conda-forge
idna                      3.3                pyhd3eb1b0_0  
menuinst                  1.4.18           py39h59b6b97_0  
nodeenv                   1.7.0              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1q               h8ffe710_0    conda-forge
pip                       21.2.4           py39haa95532_0  
platformdirs              2.5.2              pyhd8ed1ab_1    conda-forge
powershell_shortcut       0.0.1                         3  
pre-commit                2.20.0           py39hcbf5309_0    conda-forge
pycosat                   0.6.3            py39h2bbff1b_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pysocks                   1.7.1            py39haa95532_0  
python                    3.9.12               h6244533_0  
python_abi                3.9                      2_cp39    conda-forge
pywin32                   302              py39h2bbff1b_2  
pyyaml                    6.0              py39hb82d6ee_4    conda-forge
requests                  2.27.1             pyhd3eb1b0_0  
ruamel_yaml               0.15.100         py39h2bbff1b_0  
setuptools                61.2.0           py39haa95532_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.63.0             pyhd3eb1b0_0  
tzdata                    2022a                hda174b7_0  
ukkonen                   1.0.1            py39h2e07f2f_2    conda-forge
urllib3                   1.26.8             pyhd3eb1b0_0  
vc                        14.2                 h21ff451_1  
virtualenv                20.16.5          py39hcbf5309_0    conda-forge
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
win_inet_pton             1.1.0            py39haa95532_0  
wincertstore              0.2              py39haa95532_2  
yaml                      0.2.5                he774522_0

This was run inside a powershell at the root folder of the project, using the base or no environment.

xhochy · 2022-10-28T14:48:16Z

This issue here is that the Python is your base environment probably doesn't have the necessary bugfix. As reported by conda list is from the main channel and not from conda-forge. Can you try to install conda-forge::python=3.9.13 in your base environment and recheck whether pre-commit works?

master117 · 2022-11-02T12:39:56Z

@xhochy That actually worked, thank you very much.

This should be resolved by using the latest Python 3.9 and 3.10 Python builds. No need for workarounds anymore.

Maybe this should be rephrased to this works with the latest Python builds FROM conda forge

lorenzwalthert mentioned this issue Apr 10, 2020

Allow to choose installation environment lorenzwalthert/precommit#146

Closed

asottile mentioned this issue Jul 17, 2020

VS Code error: pre-commit not found. Did you forget to activate your virtualenv? pre-commit/pre-commit#1537

Closed

xhochy closed this as completed Dec 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conda installed pre-commit causes problems with graphical git front-ends #9

Conda installed pre-commit causes problems with graphical git front-ends #9

QuantCo-mr-T commented Feb 14, 2020

QuantCo-mr-T commented Feb 14, 2020

nicoddemus commented Feb 14, 2020 •

edited

Loading

gqmelo commented Feb 15, 2020

QuantCo-mr-T commented Feb 17, 2020

lorenzwalthert commented Apr 10, 2020 •

edited

Loading

lorenzwalthert commented Apr 19, 2020

kylemcmearty commented Sep 16, 2021 •

edited

Loading

Change generated pre-commit files to use `python.bat` wrapper

Change generated pre-commit files to use `python.bat` wrapper

kylemcmearty commented Sep 16, 2021 •

edited

Loading

xhochy commented Dec 30, 2021

master117 commented Oct 19, 2022

xhochy commented Oct 19, 2022

master117 commented Oct 19, 2022 •

edited

Loading

xhochy commented Oct 20, 2022

master117 commented Oct 21, 2022

xhochy commented Oct 28, 2022

master117 commented Nov 2, 2022

Conda installed pre-commit causes problems with graphical git front-ends #9

Conda installed pre-commit causes problems with graphical git front-ends #9

Comments

QuantCo-mr-T commented Feb 14, 2020

QuantCo-mr-T commented Feb 14, 2020

nicoddemus commented Feb 14, 2020 • edited Loading

gqmelo commented Feb 15, 2020

Use git from conda-forge + wrappers

Create wrappers for your current git installation

Change generated pre-commit files to use python.bat wrapper

QuantCo-mr-T commented Feb 17, 2020

lorenzwalthert commented Apr 10, 2020 • edited Loading

lorenzwalthert commented Apr 19, 2020

kylemcmearty commented Sep 16, 2021 • edited Loading

Change generated pre-commit files to use python.bat wrapper

Change generated pre-commit files to use python.bat wrapper

kylemcmearty commented Sep 16, 2021 • edited Loading

xhochy commented Dec 30, 2021

master117 commented Oct 19, 2022

xhochy commented Oct 19, 2022

master117 commented Oct 19, 2022 • edited Loading

xhochy commented Oct 20, 2022

master117 commented Oct 21, 2022

xhochy commented Oct 28, 2022

master117 commented Nov 2, 2022

nicoddemus commented Feb 14, 2020 •

edited

Loading

Change generated pre-commit files to use `python.bat` wrapper

lorenzwalthert commented Apr 10, 2020 •

edited

Loading

kylemcmearty commented Sep 16, 2021 •

edited

Loading

Change generated pre-commit files to use `python.bat` wrapper

Change generated pre-commit files to use `python.bat` wrapper

kylemcmearty commented Sep 16, 2021 •

edited

Loading

master117 commented Oct 19, 2022 •

edited

Loading