Install dependancies in virtualenv without adding it to pyproject.toml file

[antoine-gallix]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

My use case for my library is the following:

I have dependancies that I add with poetry add <...>. I have dev dependancies, that are required to run the tests, that I add with poetry add --dev <...>.

Then there are tools that I use personnally to develop, but are not required to run tests. Examples are ipython, icecream, pdb++. Those are more personal preferencies that real requirements.

I would need a way to install manually those dependencies inside the project virtualenv, without adding them to the pyproject.toml file. Using Pipenv, I could just use pip inside the environement without affecting the Pipfile. But inside poetry shell, if I use pip, it installs packages system wide (or pyenv-wide in my case).

For example if I poetry shell and pip install ipython, it doesn't install ipython in the project's virtualenv.

[jacebrowning]

Does $ poetry run pip install ipython work?

[floer32]

I was thinking about this issue while fixing coffee this morning, for whatever reason 😅

A somewhat clean way to do this would be to support nested keys under dev-dependencies ... Specifically the 'nested subtables' syntax of TOML. https://github.com/toml-lang/toml#array-of-tables

Default when doing poetry install --dev would be to ignore the nested keys, but then there'd be some way to opt-in to the other bits.

This would accomplish the optional aspect, while also offering a cleaner way than entirely-untracked dev dependencies (so it can help with tracking known-good total snapshots of the pyproject.toml that people on your team / repo are using)

[antoine-gallix]

@hangtwenty My goal is to not leave any trace at all in the pyproject.toml file, but still install packages in the virtualenv

[antoine-gallix]

@jacebrowning

Does $ poetry run pip install ipython work?

# create a new project
❯ poetry new plic
Created package plic in plic
❯ cd plic
❯ ls
plic/  pyproject.toml  README.rst  tests/

# ipython seems to call my system ipython if not in virtualenv, ok
❯ which ipython
/home/antoine/.pyenv/shims/ipython

# ipython seems to call my system ipython also when in virtualenv, ok, it's falling back on system packages
❯ poetry run which ipython
/home/antoine/.pyenv/shims/ipython

# try to install ipython in the project virtualenv
❯ poetry run pip install ipython
Collecting ipython
  Using cached https://files.pythonhosted.org/packages/14/3b/3fcf422a99a04ee493e6a4fc3014e3c8ff484a7feed238fef68bdc285085/ipython-7.3.0-py3-none-any.whl
  <...lots of install output...>
  Successfully installed backcall-0.1.0 decorator-4.3.2 ipython-7.3.0 ipython-genutils-0.2.0 jedi-0.13.3 parso-0.3.4 pexpect-4.6.0 pickleshare-0.7.5 prompt-toolkit-2.0.9 ptyprocess-0.6.0 pygments-2.3.1 six-1.12.0 traitlets-4.3.2 wcwidth-0.1.7

# ipython seems to be accessible using `poetry run` prefix
❯ poetry run which ipython
/home/antoine/.cache/pypoetry/virtualenvs/plic-py3.7/bin/ipython

# from inside poetry shell, yet, I'm getting back system ipython
❯ poetry shell
Spawning shell within /home/antoine/.cache/pypoetry/virtualenvs/plic-py3.7
# inside venv shell
plic-py3.7 ❯ which ipython
/home/antoine/.pyenv/shims/ipython

# exit poetry shell
❯ exit
❯ source source ~/.cache/pypoetry/virtualenvs/plic-py3.7/bin/activate.fish
plic-py3.7 ❯ which ipython
/home/antoine/.cache/pypoetry/virtualenvs/plic-py3.7/bin/ipython

Looks like the shell is not correctly activated with fish shell

For info, the equivalent sequence of actions works as expected in Pipenv

[floer32]

@antoine-gallix

@hangtwenty My goal is to not leave any trace at all in the pyproject.toml file, but still install packages in the virtualenv

My subjective understanding is that 'the Poetry way' would want it to be reflected in the pyproject.toml. Maybe...

[antoine-gallix]

pyproject.toml is a version controlled file that I share with my team. It's a place for project dependencies, with a section for packages that are required for running test. Everything that is imported from anywhere in the code or tests. But I don't want to put there tools that I use for debugging. Those are personal preferences that don't need to be in any requirement files, yet I want them to be present in the virtualenv so I can use them. But from my above experiment, it seems that it's the poetry shell command that is not activating the virtualenv correctly with the fish shell that I am using. After poetry shell, pip still refers to the system pip and not the virtualenv's pip. Same for ipython That doesn't happen if I manually activate the virtualenv with source /path/to/virtualenv/bin/activate.fish

[floer32]

@antoine-gallix there are cases for either side of that argument given how dependencies can interact. But overall it's probably ideal to find a solution that does not require a TOML change, agreed.

[floer32]

So I think the real issue is probably just the $PATH variable. If you installed/hooked-up pyenv before poetry, it's probably being added to $PATH before poetry things are.

If I uninstall ipython at system level and delete $HOME/.pyenv/shims/ipython then inside of poetry shell it makes which ipython work as expected. (caveat: I'm not using fish, but I was able to reproduce your issue in my environment (using zsh), and then the issue was resolved by this.

---

I think this issue becomes about is the difference between poetry run and poetry shell.
It seems ideal for Poetry to ensure these behave the same (or as close as possible) ... Or at least to provide some configuration over shell (I opened another ticket #947 about being able to explicitly choose which shell to use, but maybe it should allow further control).

The main effect we are talking about here is the impact to $PATH but I think the cause is that run does not source rc files again, whereas shell does.

[floer32]

@antoine-gallix could you run similar commands to the below, and we can see if we are reproducing the same thing?

❯ poetry run python -c "import os; print os.environ['PATH']"
/Users/michael/Library/Caches/pypoetry/virtualenvs/foo-Bp2VMsrx-py2.7/bin:/Users/michael/.pyenv/versions/2.7.14/bin:/usr/local/Cellar/pyenv/1.2.3/libexec:/usr/local/opt/[email protected]/bin:/Users/michael/.local/bin:/Users/michael/.pyenv/shims:/Users/michael/.poetry/bin [... snipped ...]

❯ poetry shell
Spawning shell within /Users/michael/Library/Caches/pypoetry/virtualenvs/foo-Bp2VMsrx-py2.7

(foo-Bp2VMsrx-py2.7) ❯ python -c "import os; print(os.environ['PATH'])"
/Users/michael/.pyenv/versions/2.7.14/bin:/usr/local/Cellar/pyenv/1.2.3/libexec:/usr/local/opt/[email protected]/bin:/Users/michael/.local/bin:/Users/michael/.pyenv/shims:/Users/michael/.poetry/bin:/usr/local/opt/node@6/bin:/Users/michael/Library/Caches/pypoetry/virtualenvs/foo-Bp2VMsrx-py2.7/bin:/Users/michael/.pyenv/versions/2.7.14/bin:/usr/local/Cellar/pyenv/1.2.3/libexec:/usr/local/opt/[email protected]/bin:/Users/michael/.local/bin:/Users/michael/.pyenv/shims:/Users/michael/.poetry/bin #[... snipped ...]

[antoine-gallix]

seems related to #571

[floer32]

@antoine-gallix haha I was just about to post that! We were looking at the same time

[floer32]

Related to #497 too (which may/maynot get closed in favor of solutions discussed on #571 ...?)

But already can say, proper solution is not only about $PATH

#497 (comment) 👇

Heads up, stay away from PATH and the pyenv model until some shim problems are resolved, see pyenv/pyenv#1112

In a similar theme of avoiding PATH, the latest releases of miniconda no longer modify the ~/.bashrc to add conda to the PATH. The latest releases use something like an /etc/profile.d/conda.sh to load bash functions (check it with type conda for example).

[antoine-gallix]

OK so to summarize, my problem comes from the fact that the pyenv shell command is not working on my system. Regarding dependencies, it's all good, but python and pip when shell is activated call the system/pyenv commands instead of the venv ones. My workaround is to make a custom poetry shell command in fish.

# works with the latest 1.0.0 version of poetry
function poetry_shell
    source (poetry env info -p)/bin/activate.fish
end

From there I can install things in the venv with pip without affecting dependencies files used by poetry.

[floer32]

Pasting a post for posterity, regarding virtualenv and $PATH. (doesn't cover pyenv specifically but one can generalize and understand how some things may be sensitive to interactions)

https://pythonspeed.com/articles/activate-virtualenv-dockerfile/

[pmav99]

From there I can install things in the venv with pip without affecting dependencies files used by poetry.

You might still install a package with confliciting dependencies though. Perhaps it would make sense to have a special mode of poetry add that will resolve dependencies and install a package without adding it to pyproject.toml.

[antoine-gallix]

True, that would be a nice feature. Something like poetry add --extra.

[richin13]

Or poetry add --no-save

[pmav99]

Continuing the discussion from here

You might still install a package with confliciting dependencies though. Perhaps it would make sense to have a special mode of poetry add that will resolve dependencies and install a package without adding it to pyproject.toml.

The semantics of this use case seem weird and difficult. If the "temporary" package has overlapping dependencies with non-temporary packages, what happens? Does it change the versions of the common dependencies? If so, does it update the lock file? Does it instead just fail outright?

I think that the only sensible action for conflicting dependencies is to fail loudly. If someones wants to install the package anyway, he can always use plain pip install ....

The OP on that issue says:

Then there are tools that I use personnally to develop, but are not required to run tests. Examples are ipython, icecream, pdb++. Those are more personal preferencies that real requirements.

In node, the accepted solution for this use-case is either (1) install it globally since it's not relevant to the project at hand or (2) install it as part of the project because you use it to develop the project (and hence it is a dev dependency).

• Installing globally is not always a good or even a possible solution.
• Adding something like e.g. pudb as a dev dependency really feels like abusing the dev dependencies. I might like text based debuggers, but I don't see why I need to force other developers to install it. (e.g. a PyCharm user would probably prefer to use the integrated debugger instead of installing a 3rd party one).

It seems to me that trying to support a third, middle road is likely to cause more confusion than help.

Trying to keep things simple is indeed an important goal. That being said, I think that this is a valid and not that uncommon usecase + adding an extra flag to poetry add would not complicate matters that much.

[seansfkelley]

I think that the only sensible action for conflicting dependencies is to fail loudly. If someones wants to install the package anyway, he can always use plain pip install ....

I agree, though at that point I don't see what value Poetry is adding if any conflict leads to failure and it's specifically not locking dependencies. I would be concerned about mixed messaging about what Poetry is actually doing, though whether it actually is mixed messaging would depend on the resolution of #648, i.e., if Poetry defaults/prefers to remove dependencies not in the lock file then it seems like a conflicting feature to support installing things without locking them in the same tool.

Installing globally is not always a good or even a possible solution.

• "good": what do you mean by that? If I use a tool for my own development flow I probably use it across all projects in that language. My globally installed dependencies don't need to be in tip-top shape, in fact, I kind of treat it as a dumping ground since every other project is in its own managed, isolated environment.
• "possible": is that a Python 2/3 thing, or did you mean something else?

[pmav99]

The problem is that if you just blindly pip install some tool with conflicting dependencies you will break your application/library. The added value of using poetry to resolve the dependencies before installing it are:

1. If there are no conflicting dependencies, you will know this and you will be confident that everything will be working fine.
2. If there are conflicting dependencies, you will know this without breaking your application's virtualenv.

"good": what do you mean by that?

Depending on how you do it, Installing things globally might break other stuff/tools.

"possible": is that a Python 2/3 thing, or did you mean something else?

Some tools, e.g. pudb need to be installed in the same virtualenv as your code.

[shonfeder]

I can’t tell from this issue what the recommended approach is for this use case. It looks like the author of the issue closed it because they found a workaround involving a shell alias, but is that still the recommended approach? Thanks Amin advance for any guidance here!

[nitsanh]

python -c "import os; print(os.environ['PATH'])"

I'm running with poetry 1.3.2 and running poetry shell, then pip install ... works fine, it installs the package inside poetry's venv, and doesn't add anything to pyproject.toml

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.