environment variables not assigned via conda activate

[xylar]

Details

• Read the Docs project URL: https://github.com/MPAS-Dev/MPAS-Analysis
• Build URL (if applicable): https://readthedocs.org/projects/mpas-analysis/builds/8631850/
• Read the Docs username (if applicable): xylar

Expected Result

from mpl_toolkits.basemap import Basemap (basemap one of the dependencies of my project) would happen without any errors, allowing autosummary to parse my docstrings and create the autogenerate API.

Actual Result

...
    from mpl_toolkits.basemap import Basemap
  File "/home/docs/checkouts/readthedocs.org/user_builds/mpas-analysis/conda/develop/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 155, in <module>
    pyproj_datadir = os.environ['PROJ_LIB']
  File "/home/docs/checkouts/readthedocs.org/user_builds/mpas-analysis/conda/develop/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'PROJ_LIB'

The problem seems to be that readthedocs uses absolute paths within the conda environment but doesn't ever call conda activate ${evironment}. As a result, conda packages do not set environment variables (which would normally happen as part of activate and other packages fail because the environment variables are not present. In this case, proj4 should set $PROJ_LIB, which basemap expects to find.

See conda-forge/basemap-feedstock#30 for a lengthy discussion of this issue

[stsewd]

Thanks, I think I've seen this issue before with other projects, I'll try to figure out how we can support conda activate (we don't share env variables between commands I think). Also, we do support setting env variables manually, maybe you can try hardcoding these? https://docs.readthedocs.io/en/stable/guides/environment-variables.html

[xylar]

Thanks @stsewd, yes, I was able to solve the problem by manually setting PROJ_LIB in my conf.py. I think that's okay for now since basemap is the only package I'm aware of that uses environment variables from another package. But I suspect this problem could balloon going forward if more packages decide to use this functionality. RTD is certainly not the only place where this is causing issues, though, so perhaps conda and conda-forge will hesitate to use this functionality.

[WardBrian]

I'd like to echo once again that this would be great to address, or at the very least document the current behavior. It has given me quite a hard time to figure out why my conda environment didn't seem to be the same as when I did it locally, and ultimately it was the lack of setup done by conda-activate

[valeriupredoi]

we have a similar problem with a package that needs an env variable be set so it can find the file that contains package-level settings (libs, flags etc), see ESMValGroup/ESMValCore#2001 - the variable in cause is set at conda activate [env] point, and is a dynamical variable in that the file's location is eg ~/miniconda3/envs/ENV/lib/esmf.mk - if a conda activate ... is not performed, then there is no way to access the ENV's name since that's either latest, stable or the number of the PR that triggers the RTD build - any way to go around this, please? 🍺

[zklaus]

An alternative could be to use conda run. One would somehow have to prepend conda run --name <envname> to all commands run in an Conda environment.

[humitos]

Is this still an issue? Is there any workaround we can document? I don't think we will change the way that we are running conda on Read the Docs; we are not experts on Conda but we haven't had issues with it lately and I would try to avoid changing the main command that runs the Conda because I'm sure it will break some people's projects. However, I'd like to document any workaround we have here or just close the issue if we don't have an immediate and simple solution.

[valeriupredoi]

hi @humitos indeed this is still an issue - but it's inherent of the way things are done on Readthedocs, and if you guys don't want to change them it's okay - we found a workaround this thanks to @zklaus - conda variables are loaded via hot-wiring the main conda executable via conda run:

build:
  os: ubuntu-22.04
  tools:
    python: "mambaforge-4.10"
  jobs:
    post_create_environment:
      # use conda run executable wrapper to have all env variables
      - conda run -n ${CONDA_DEFAULT_ENV} pip install . --no-deps

Cheers 🍺

[humitos]

@valeriupredoi thanks for sharing this chunk of YAML here. I'm sure it will be pretty useful to other users as well 💯

[RaulPPelaez]

When running commands using "conda run", environment variables like CONDA_PREFIX are not available.
I have a project that is built and installed using CMake, I have not found a better workaround than just hardcoding the path in my .readthedocs.yml:

version: 2

submodules:
  include: all
  recursive: true

build:
  os: "ubuntu-22.04"
  tools:
     python: "mambaforge-22.9"
  commands:
    - mkdir build
    - mamba env create --file environment.yml
    - cd build && mamba run -n libmobility cmake -DCMAKE_INSTALL_PREFIX=/home/docs/checkouts/readthedocs.org/user_builds/libmobility/conda/libmobility .. && mamba run -n libmobility make -j4 all install
    - cd docs && mamba run -n libmobility make html
    - mkdir -p $READTHEDOCS_OUTPUT/
    - mv ./docs/build/html $READTHEDOCS_OUTPUT/html

Is there maybe some preferred place to install? Some environment variable perhaps?
There is READTHEDOCS_VIRTUALENV_PATH , but the documentation specifically says that it is not available when using conda.

[humitos]

Some environment variable perhaps?
There is READTHEDOCS_VIRTUALENV_PATH , but the documentation specifically says that it is not available when using conda.

I think the path you have harcoded could be replaced by CONDA_ENVS_PATH. It's an undocumented variable tho. I should be documented as READTHEDOCS_CONDA_ENV_PATH eventually I suppose.