Support virtual environment (#578)

* Support virtual environment

* Add test_venv.jl

* Add python3.4-venv in Travis

* Fix pythonhome_of for Julia 0.6

* Properly treat venv path in Windows tests

Since this is useful outside PyCall testing, I included this
functionality in python_cmd function instead of inlining it to the
test.

* Mark venv test broken in Windows

* Support venv properly

* `venv` support is now properly tested.  This finds a bug in
  `pythonhome_of`: `sys.base_prefix` should be used instead of
  `sys.prefix` (same for `sys.base_exec_prefix`).

* The path to Python executable passed to `pythonhome_of` was ignored
  when `python_cmd` was introduced.  This is fixed now.

* Use `python_cmd` in `find_libpython`.  This properly sets
  `PYTHONIOENCODING`.

* Test with virtualenv command as well

* Specify Python version in virtualenv activation test

* Move pythonhome_of to depsutils.jl and use it during build

* Support "venv activation" test inside virtualenv

* Show correct Python executable when import fails

* Use static memory location than malloc

* Create virtual environment at non-ascii evil path

* Be kind to Python 2

* Unify Py_SetPythonHome wrapper functions

* reinterpret source array instead of dest

* Improve docs and comments

* Simplify __init__() by merging venv code path

* Stop using Compat in test_venv.jl

as in 45cbcee

* Use property-based syntax in test_venv.jl

* Add docs on virtual environment usage

* Remove unnecessary VERSION < v"0.7.0-" branch

* grammar fixes; rm references to internal details about Python API functions

* inline links

.travis.yml

@@ -1,4 +1,5 @@
 language: julia
+dist: trusty  # update "python3.4-venv" below when updating
 os:
     - linux
 julia:
@@ -10,6 +11,7 @@ addons:
     packages:
       - python-numpy
       - python3-numpy
+      - python3.4-venv  # Ubuntu Trusty 14.04 does not have python3-venv
 env:
   global:
     - PYCALL_DEBUG_BUILD="yes"

README.md

@@ -75,10 +75,12 @@ unless you set the `PYTHON` environment variable or delete the file
 
 **Note:** If you use Python
 [virtualenvs](http://docs.python-guide.org/en/latest/dev/virtualenvs/),
-then be aware that PyCall *uses the virtualenv it was built with*, even
-if you switch virtualenvs.  If you want to switch PyCall to use a
-different virtualenv, then you should switch virtualenvs and run
-`rm(Pkg.dir("PyCall","deps","PYTHON")); Pkg.build("PyCall")`.
+then be aware that PyCall *uses the virtualenv it was built with* by
+default, even if you switch virtualenvs.  If you want to switch PyCall
+to use a different virtualenv, then you should switch virtualenvs and
+run `rm(Pkg.dir("PyCall","deps","PYTHON")); Pkg.build("PyCall")`.
+Alternatively, see [Python virtual environments](#python-virtual-environments)
+section below for switching virtual environment at run-time.
 
 **Note:** Usually, the necessary libraries are installed along with
 Python, but [pyenv on MacOS](https://github.com/JuliaPy/PyCall.jl/issues/122)
@@ -596,6 +598,35 @@ and so on.
 Here, instead of `pyimport`, we have used the function `pyimport_conda`.   The second argument is the name of the [Anaconda package](https://docs.continuum.io/anaconda/pkg-docs) that provides this module.   This way, if importing `scipy.optimize` fails because the user hasn't installed `scipy`, it will either (a) automatically install `scipy` and retry the `pyimport` if PyCall is configured to use the [Conda](https://github.com/Luthaf/Conda.jl) Python install (or
 any other Anaconda-based Python distro for which the user has installation privileges), or (b) throw an error explaining that `scipy` needs to be installed, and explain how to configure PyCall to use Conda so that it can be installed automatically.   More generally, you can call `pyimport(module, package, channel)` to specify an optional Anaconda "channel" for installing non-standard Anaconda packages.
 
+## Python virtual environments
+
+Python virtual environments created by [`venv`](https://docs.python.org/3/library/venv.html) and [`virtualenv`](https://virtualenv.pypa.io/en/latest/)
+can be used from `PyCall`, *provided that the Python executable used
+in the virtual environment is linked against the same `libpython` used
+by `PyCall`*.  Note that virtual environments created by `conda` are not
+supported.
+
+To use `PyCall` with a certain virtual environment, set the environment
+variable `PYCALL_JL_RUNTIME_PYTHON` *before* importing `PyCall` to
+path to the Python executable.  For example:
+
+```julia
+$ source PATH/TO/bin/activate  # activate virtual environment in system shell
+
+$ julia  # start Julia
+...
+
+julia> ENV["PYCALL_JL_RUNTIME_PYTHON"] = Sys.which("python")
+"PATH/TO/bin/python3"
+
+julia> using PyCall
+
+julia> pyimport("sys").executable
+"PATH/TO/bin/python3"
+```
+Similarly, the `PYTHONHOME` path can be changed by the environment variable
+`PYCALL_JL_RUNTIME_PYTHONHOME`.
+
 ## Author
 
 This package was written by [Steven G. Johnson](http://math.mit.edu/~stevenj/).

deps/build.jl

@@ -12,28 +12,6 @@ struct UseCondaPython <: Exception end
 
 #########################################################################
 
-# Fix the environment for running `python`, and setts IO encoding to UTF-8.
-# If cmd is the Conda python, then additionally removes all PYTHON* and
-# CONDA* environment variables.
-function pythonenv(cmd::Cmd)
-    env = copy(ENV)
-    if dirname(cmd.exec[1]) == abspath(Conda.PYTHONDIR)
-        pythonvars = String[]
-        for var in keys(env)
-            if startswith(var, "CONDA") || startswith(var, "PYTHON")
-                push!(pythonvars, var)
-            end
-        end
-        for var in pythonvars
-            pop!(env, var)
-        end
-    end
-    # set PYTHONIOENCODING when running python executable, so that
-    # we get UTF-8 encoded text as output (this is not the default on Windows).
-    env["PYTHONIOENCODING"] = "UTF-8"
-    setenv(cmd, env)
-end
-
 pyvar(python::AbstractString, mod::AbstractString, var::AbstractString) = chomp(read(pythonenv(`$python -c "import $mod; print($mod.$var)"`), String))
 
 pyconfigvar(python::AbstractString, var::AbstractString) = pyvar(python, "distutils.sysconfig", "get_config_var('$var')")
@@ -117,11 +95,6 @@ include("depsutils.jl")
 
 #########################################################################
 
-# A couple of key strings need to be stored as constants so that
-# they persist throughout the life of the program.  In Python 3,
-# they need to be wchar_t* data.
-wstringconst(s) = string("Base.cconvert(Cwstring, \"", escape_string(s), "\")")
-
 # we write configuration files only if they change, both
 # to prevent unnecessary recompilation and to minimize
 # problems in the unlikely event of read-only directories.
@@ -197,15 +170,7 @@ try # make sure deps.jl file is removed on error
     # Get PYTHONHOME, either from the environment or from Python
     # itself (if it is not in the environment or if we are using Conda)
     PYTHONHOME = if !haskey(ENV, "PYTHONHOME") || use_conda
-        # PYTHONHOME tells python where to look for both pure python
-        # and binary modules.  When it is set, it replaces both
-        # `prefix` and `exec_prefix` and we thus need to set it to
-        # both in case they differ. This is also what the
-        # documentation recommends.  However, they are documented
-        # to always be the same on Windows, where it causes
-        # problems if we try to include both.
-        exec_prefix = pysys(python, "exec_prefix")
-        Sys.iswindows() ? exec_prefix : pysys(python, "prefix") * ":" * exec_prefix
+        pythonhome_of(python)
     else
         ENV["PYTHONHOME"]
     end
@@ -223,10 +188,8 @@ try # make sure deps.jl file is removed on error
     const python = "$(escape_string(python))"
     const libpython = "$(escape_string(libpy_name))"
     const pyprogramname = "$(escape_string(programname))"
-    const wpyprogramname = $(wstringconst(programname))
     const pyversion_build = $(repr(pyversion))
     const PYTHONHOME = "$(escape_string(PYTHONHOME))"
-    const wPYTHONHOME = $(wstringconst(PYTHONHOME))
 
     "True if we are using the Python distribution in the Conda package."
     const conda = $use_conda

deps/depsutils.jl

@@ -13,21 +13,143 @@ function findsym(lib, syms...)
     error("no symbol found from: ", syms)
 end
 
+# Static buffer to make sure the string passed to libpython persists
+# for the lifetime of the program, as CPython API requires:
+const __buf_programname = Vector{UInt8}(undef, 1024)
+const __buf_pythonhome = Vector{UInt8}(undef, 1024)
+
 # Need to set PythonHome before calling GetVersion to avoid warning (#299).
 # Unfortunately, this poses something of a chicken-and-egg problem because
 # we need to know the Python version to set PythonHome via the API.  Note
 # that the string (or array) passed to Py_SetPythonHome needs to be a
 # constant that lasts for the lifetime of the program, which is why we
-# can't use Cwstring here (since that creates a temporary copy).
-function Py_SetPythonHome(libpy, PYTHONHOME, wPYTHONHOME, pyversion)
-    if !isempty(PYTHONHOME)
-        if pyversion.major < 3
-            ccall(Libdl.dlsym(libpy, :Py_SetPythonHome), Cvoid, (Cstring,), PYTHONHOME)
-        else
-            ccall(Libdl.dlsym(libpy, :Py_SetPythonHome), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)
-        end
+# prepare static buffer __buf_pythonhome, copy the string to it, and then
+# pass the pointer to the buffer to the CPython API.
+function Py_SetPythonHome(libpy, pyversion, PYTHONHOME::AbstractString)
+    isempty(PYTHONHOME) && return
+    if pyversion.major < 3
+        ccall(Libdl.dlsym(libpy, :Py_SetPythonHome), Cvoid, (Cstring,),
+              _preserveas!(__buf_pythonhome, Cstring, PYTHONHOME))
+    else
+        ccall(Libdl.dlsym(libpy, :Py_SetPythonHome), Cvoid, (Ptr{Cwchar_t},),
+              _preserveas!(__buf_pythonhome, Cwstring, PYTHONHOME))
+    end
+end
+
+function Py_SetProgramName(libpy, pyversion, programname::AbstractString)
+    isempty(programname) && return
+    if pyversion.major < 3
+        ccall(Libdl.dlsym(libpy, :Py_SetProgramName), Cvoid, (Cstring,),
+              _preserveas!(__buf_programname, Cstring, programname))
+    else
+        ccall(Libdl.dlsym(libpy, :Py_SetProgramName), Cvoid, (Ptr{Cwchar_t},),
+              _preserveas!(__buf_programname, Cwstring, programname))
     end
 end
 
+"""
+    _preserveas!(dest::Vector{UInt8}, (Cstring|Cwstring), x::String) :: Ptr
+
+Copy `x` as `Cstring` or `Cwstring` to `dest` and return a pointer to
+`dest`.  Thus, this pointer is safe to use as long as `dest` is
+protected from GC.
+"""
+function _preserveas!(dest::Vector{UInt8}, ::Type{Cstring}, x::AbstractString)
+    s = transcode(UInt8, String(x))
+    copyto!(dest, s)
+    dest[length(s) + 1] = 0
+    return pointer(dest)
+end
+
+function _preserveas!(dest::Vector{UInt8}, ::Type{Cwstring}, x::AbstractString)
+    s = Base.cconvert(Cwstring, x)
+    copyto!(dest, reinterpret(UInt8, s))
+    return pointer(dest)
+end
+
+
 # need to be able to get the version before Python is initialized
 Py_GetVersion(libpy) = unsafe_string(ccall(Libdl.dlsym(libpy, :Py_GetVersion), Ptr{UInt8}, ()))
+
+# Fix the environment for running `python`, and setts IO encoding to UTF-8.
+# If cmd is the Conda python, then additionally removes all PYTHON* and
+# CONDA* environment variables.
+function pythonenv(cmd::Cmd)
+    env = copy(ENV)
+    if dirname(cmd.exec[1]) == abspath(Conda.PYTHONDIR)
+        pythonvars = String[]
+        for var in keys(env)
+            if startswith(var, "CONDA") || startswith(var, "PYTHON")
+                push!(pythonvars, var)
+            end
+        end
+        for var in pythonvars
+            pop!(env, var)
+        end
+    end
+    # set PYTHONIOENCODING when running python executable, so that
+    # we get UTF-8 encoded text as output (this is not the default on Windows).
+    env["PYTHONIOENCODING"] = "UTF-8"
+    setenv(cmd, env)
+end
+
+
+function pythonhome_of(pyprogramname::AbstractString)
+    if Sys.iswindows()
+        # PYTHONHOME tells python where to look for both pure python
+        # and binary modules.  When it is set, it replaces both
+        # `prefix` and `exec_prefix` and we thus need to set it to
+        # both in case they differ. This is also what the
+        # documentation recommends.  However, they are documented
+        # to always be the same on Windows, where it causes
+        # problems if we try to include both.
+        script = """
+        import sys
+        if hasattr(sys, "base_exec_prefix"):
+            sys.stdout.write(sys.base_exec_prefix)
+        else:
+            sys.stdout.write(sys.exec_prefix)
+        """
+    else
+        script = """
+        import sys
+        if hasattr(sys, "base_exec_prefix"):
+            sys.stdout.write(sys.base_prefix)
+            sys.stdout.write(":")
+            sys.stdout.write(sys.base_exec_prefix)
+        else:
+            sys.stdout.write(sys.prefix)
+            sys.stdout.write(":")
+            sys.stdout.write(sys.exec_prefix)
+        """
+        # https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHOME
+    end
+    return read(pythonenv(`$pyprogramname -c $script`), String)
+end
+# To support `venv` standard library (as well as `virtualenv`), we
+# need to use `sys.base_prefix` and `sys.base_exec_prefix` here.
+# Otherwise, initializing Python in `__init__` below fails with
+# unrecoverable error:
+#
+#   Fatal Python error: initfsencoding: unable to load the file system codec
+#   ModuleNotFoundError: No module named 'encodings'
+#
+# This is because `venv` does not symlink standard libraries like
+# `virtualenv`.  For example, `lib/python3.X/encodings` does not
+# exist.  Rather, `venv` relies on the behavior of Python runtime:
+#
+#   If a file named "pyvenv.cfg" exists one directory above
+#   sys.executable, sys.prefix and sys.exec_prefix are set to that
+#   directory and it is also checked for site-packages
+#   --- https://docs.python.org/3/library/venv.html
+#
+# Thus, we need point `PYTHONHOME` to `sys.base_prefix` and
+# `sys.base_exec_prefix`.  If the virtual environment is created by
+# `virtualenv`, those `sys.base_*` paths point to the virtual
+# environment.  Thus, above code supports both use cases.
+#
+# See also:
+# * https://docs.python.org/3/library/venv.html
+# * https://docs.python.org/3/library/site.html
+# * https://docs.python.org/3/library/sys.html#sys.base_exec_prefix
+# * https://github.com/JuliaPy/PyCall.jl/issues/410

src/PyCall.jl

@@ -36,6 +36,12 @@ import Base.Iterators: filter
 include(joinpath(dirname(@__FILE__), "..", "deps","depsutils.jl"))
 include("startup.jl")
 
+"""
+Python executable used by PyCall in the current process.
+"""
+current_python() = _current_python[]
+const _current_python = Ref(pyprogramname)
+
 #########################################################################
 
 # Mirror of C PyObject struct (for non-debugging Python builds).
@@ -506,7 +512,7 @@ you want to use, run Pkg.build("PyCall"), and re-launch Julia.
                     msg = msg * """
 PyCall is currently configured to use the Python version at:
 
-$python
+$(current_python())
 
 and you should use whatever mechanism you usually use (apt-get, pip, conda,
 etcetera) to install the Python package containing the $name module.