Set PATH before running conda #121

tkf · 2018-09-20T09:16:25Z

close #125

stevengj · 2018-09-20T11:53:26Z

src/Conda.jl

@@ -103,6 +103,8 @@ function _set_conda_env(cmd, env::Environment=ROOTENV)
    env_var["PYTHONIOENCODING"]="UTF-8"
    env_var["CONDARC"] = conda_rc(env)
    env_var["CONDA_PREFIX"] = prefix(env)
+    path_sep = Compat.Sys.iswindows() ? ';' : ':'
+    env_var["PATH"] = bin_dir(env) * path_sep * ENV["PATH"]


get(ENV, "PATH", "")

stevengj · 2018-09-20T11:55:14Z

test/runtests.jl

+
+cmd = `python -c "import sys; print(sys.executable)"`
+path = rstrip(read(_set_conda_env(cmd, env), String))
+@test startswith(normpath(path), normpath(bin_dir(env)))


startwith or normpath(dirname(path)) == normpath(bin_dir(env))?

stevengj · 2018-09-20T11:57:21Z

test/runtests.jl

+    @test isfile(pythonpath)
+
+    cmd = Conda._set_conda_env(`$pythonpath -c "import zmq"`, env)
+    @test_throws Exception run(cmd)


This won't work if pyzmq is already installed — people don't necessarily run test Conda on a clean installation.

In particular, what happens if you run the tests twice in a row? Won't pyzmq still be installed from the previous run?

Oops, @test !success(cmd) was just a leftover from the previous version.

To make ]test Conda easier, how about doing

try Conda.rm("--all", env) catch end

(or with something similar with cleaner handling of exception)?

stevengj · 2018-09-20T12:01:16Z

I'm still unclear why it is necessary to set the PATH. Can't you just run python by specifying the full pathname?

tkf · 2018-09-20T12:18:09Z

I don't know, but from JuliaLang/IJulia.jl#739 (comment) and #119 (comment) (and that conda is not super well-behaved package manager, at least ATM), I just thought I'd try. But making the test in #122 fail properly was already hard...

tkf · 2018-09-20T12:27:12Z

BTW, when I run conda shell.powershell activate base (which is, I suppose, the internal of conda activate), I get:

$env:CONDA_DEFAULT_ENV = "base"
$env:CONDA_EXE = "/home/takafumi/miniconda3/bin/conda"
$env:CONDA_PREFIX = "/home/takafumi/miniconda3"
$env:CONDA_PROMPT_MODIFIER = "(base) "
$env:CONDA_PYTHON_EXE = "/home/takafumi/miniconda3/bin/python"
$env:CONDA_SHLVL = "1"
$env:PATH = "/home/takafumi/miniconda3\bin;..."

(the output is weird since I'm on linux)

So it seems PATH is the only relevant thing conda activate ... alters.

tkf · 2018-09-20T12:54:14Z

src/Conda.jl

@@ -103,6 +103,12 @@ function _set_conda_env(cmd, env::Environment=ROOTENV)
    env_var["PYTHONIOENCODING"]="UTF-8"
    env_var["CONDARC"] = conda_rc(env)
    env_var["CONDA_PREFIX"] = prefix(env)
+    path_sep = Compat.Sys.iswindows() ? ';' : ':'
+    if haskey(ENV, "PATH")
+        env_var["PATH"] = bin_dir(env) * path_sep * ENV["PATH"]


#121 (comment): Good catch, but get(ENV, "PATH", "") could put $PWD into $PATH?

tkf · 2018-09-20T13:12:56Z

So it turned out Windows searches PATH for a DLL https://msdn.microsoft.com/en-us/library/7d83bc18.aspx?f=255&MSPPError=-2147217396 (via ContinuumIO/anaconda-issues#10082 (comment))

tkf · 2018-09-20T13:15:17Z

But only (random?) half of the tests are fixed by this https://ci.appveyor.com/project/StevenGJohnson/conda-jl/build/1.0.121 So strange...

tkf · 2018-09-20T14:33:27Z

Reading ContinuumIO/anaconda-issues#10082, it looks like it would be fixed on conda side. Maybe it's better to do Conda.add("python=3.6") inside Conda._install_conda while waiting for that?

stevengj · 2018-09-20T18:44:06Z

I'm worried that this will also screw up PyCall… importing Python modules won't work if the Python modules depend on the PATH being set.

tkf · 2018-09-20T23:54:18Z

Yeah, Conda.jl has to mutate ENV. Or maybe PyCall.jl can temporary mutate ENV in pyimport_conda? (Similar hack for LD_LIBRARY_PATH on Linux won't work, but I don't know what happens in Windows.)

But that's ugly so Conda.add("python=3.6") started sound better to me... Or using slightly old miniconda3 URL would be the easiest solution?

This reverts a subset of the patch in 7094e2a

tkf · 2018-09-23T23:05:43Z

src/Conda.jl

+
+Extra `ENV["PATH"]` required for conda environment `env`.
+"""
+function extra_path(env::Environment)


Somebody has to write a code to pass this to SetDllDirectory or AddDllDirectory in PyCall.__init__: #125 (comment)

stevengj · 2018-09-23T23:13:52Z

src/Conda.jl

+
+function _set_path(env_var, env::Environment)
+    if Compat.Sys.iswindows()
+        # Environment variables are case-insensitive.  `Base.EnvDict`


Can you file a Julia issue for this? Seems like it should be fixed in Base.

It's just because

julia> typeof(copy(ENV)) Dict{String,String}

To solve this, I think you need to add a case-insensitive in-memory dict to Base.

Seems like the easiest fix would be to replace copy(ENV) with

env_var = Dict(uppercase(k)=>v for (k,v) in ENV)

(Otherwise the other code in _set_conda_env would have to be fixed too.)

stevengj · 2018-09-23T23:14:34Z

src/Conda.jl

+    if Compat.Sys.iswindows()
+        return join([
+            prefix(env),
+            joinpath(prefix(env), "Library", "mingw-w64", "bin"),


What if it is 32-bit Windows?

I don't know. I don't notice any branching in conda/activate.py https://github.com/conda/conda/blob/4.6.0b0/conda/activate.py (But I'm not a Python interpreter)

If the folder does not exist, wouldn't it just be ignored? Also, it looks like win-32 and win-64 are different packages for binary libraries: https://anaconda.org/anaconda/pyzmq/files So I guess it wouldn't be installed in the first place?

stevengj · 2018-09-23T23:15:34Z

src/Conda.jl

-            # * https://github.com/ContinuumIO/anaconda-issues/issues/10082
-            # * https://github.com/conda/conda/issues/7789
-            res = "https://repo.continuum.io/miniconda/Miniconda$(MINICONDA_VERSION)-4.5.4-"
-        end


Seems like this PR could be done before #125.

Well..., the id of this PR is 121 (< #124)

Joking aside, this PR needs a coordination with PyCall to use SetDllDirectory/AddDllDirectory and for IJulia to tweak setup ENV["PATH"] before launching Jupyter Notebook. So I still think it makes sense to release a quick fix with #124 first to help Windows users.

I don't know much about Windows so please feel free to close this PR and implement another solution. I re-opened this PR because I though I'd better cleanup my suboptimal solution.

stevengj · 2019-01-03T15:22:08Z

Sorry I haven't looked at this for a while. CI is failing?

tkf · 2019-01-03T20:06:02Z

I forgot the status of this PR :)

But I remember thinking that new conda version may provide a better way to do this (via conda run). I'll check if we can do this without hard-coding the paths for Windows.

tkf · 2020-02-12T02:09:22Z

It looks like #170 supersedes this PR. Closing...

tkf force-pushed the win-conda branch 3 times, most recently from 8614a51 to 57ca293 Compare September 20, 2018 10:41

tkf mentioned this pull request Sep 20, 2018

DON'T MERGE #122

Closed

tkf force-pushed the win-conda branch from 57ca293 to cfb468d Compare September 20, 2018 11:52

stevengj reviewed Sep 20, 2018

View reviewed changes

tkf force-pushed the win-conda branch from cfb468d to 0cf9602 Compare September 20, 2018 11:55

stevengj reviewed Sep 20, 2018

View reviewed changes

tkf commented Sep 20, 2018

View reviewed changes

tkf closed this Sep 21, 2018

tkf added 3 commits September 23, 2018 15:32

Set PATH before running conda

93acd8b

Unpin Miniconda3 version on Windows

57ecb2d

This reverts a subset of the patch in 7094e2a

Test Conda.add("jupyter", env)

dfdfa38

tkf reopened this Sep 23, 2018

tkf force-pushed the win-conda branch from 9cb5d04 to dfdfa38 Compare September 23, 2018 22:50

tkf commented Sep 23, 2018

View reviewed changes

stevengj reviewed Sep 23, 2018

View reviewed changes

tkf mentioned this pull request Sep 23, 2018

environment vars on Windows should be case-insensitive JuliaLang/julia#29334

Closed

Test and fix _set_path

c6260c5

stevengj mentioned this pull request Jan 3, 2019

don't require jupyter for installation JuliaLang/IJulia.jl#791

Merged

tkf mentioned this pull request Jul 5, 2019

env base not activated on Windows #119

Open

tkf closed this Feb 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set PATH before running conda #121

Set PATH before running conda #121

tkf commented Sep 20, 2018 •

edited

Loading

stevengj Sep 20, 2018

stevengj Sep 20, 2018

stevengj Sep 20, 2018

stevengj Sep 20, 2018

tkf Sep 20, 2018

stevengj commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf Sep 20, 2018

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

stevengj commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf Sep 23, 2018

stevengj Sep 23, 2018

tkf Sep 23, 2018

stevengj Jan 3, 2019 •

edited

Loading

stevengj Sep 23, 2018

tkf Sep 23, 2018

stevengj Sep 23, 2018

tkf Sep 23, 2018

stevengj commented Jan 3, 2019

tkf commented Jan 3, 2019

tkf commented Feb 12, 2020

Set PATH before running conda #121

Set PATH before running conda #121

Conversation

tkf commented Sep 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevengj commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

Choose a reason for hiding this comment

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

tkf commented Sep 20, 2018

stevengj commented Sep 20, 2018

tkf commented Sep 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevengj Jan 3, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevengj commented Jan 3, 2019

tkf commented Jan 3, 2019

tkf commented Feb 12, 2020

tkf commented Sep 20, 2018 •

edited

Loading

stevengj Jan 3, 2019 •

edited

Loading