Command killed due to excessive memory consumption

[lukelbd]

Details

• Read the Docs project URL: https://proplot.readthedocs.io/en/latest/
• Build URL (if applicable): https://readthedocs.org/projects/proplot/builds/
• Read the Docs username (if applicable): lukelbd

Expected Result

Successful build.

Actual Result

Failed build. The conda environment solve step causes excess memory consumption.

I also have python.system_packages = True in my yaml. Maybe I can reduce the environment solve complexity by using the shared packages? Currently the contents of environment.yml are:

name: proplot-dev
channels:
  - conda-forge
dependencies:
  - pip
  - numpy
  - xarray
  - pandas
  - matplotlib
  - cartopy
  - basemap
  - nbsphinx
  - sphinx
  - ipython
  - ipykernel
  - pip:
    - lxml
    - pyyaml
    - sphinx_rtd_theme
    - git+https://github.com/lukelbd/[email protected]

But I'm also worried about memory consumption for the nbsphinx steps, which generate some fairly high resolution figures. Before this merge I used nbconvert to manually generate RST files and synced those.

[humitos]

Hi @lukelbd! The main problem here is that conda consumes a lot of memory to do the package resolution. We had this problem in the past and it's a known issue in the conda community.

I enabled two flags in your project that could help with this (the most important one, is to update conda at startup, since the latest conda release has some memory consumptions improvements).

I triggered a build already. Let's see how it goes.

[humitos]

That didn't help too much. I had to increase the memory limit to 2.5Gb to make the conda create env step to pass.

The build is still failing because your proplot library is not properly installed in the conda environment created, so it failed when importing it from Sphinx. See https://readthedocs.org/projects/proplot/builds/10046893/

[lukelbd]

Thanks a lot!

I fixed the failed build by adding proplot to environment.yml with a

dependencies:
  - pip:
    .. # relative path for installing current proplot master branch

But now it fails because I've renamed some classes and class methods. I think it will run if I can call make clean on the RTD server. How can I do this?

[lukelbd]

It seems this error is related to #208 (which was posted by the astropy developers). Astropy released the little-known sphinx-automodapi plugin that I also use for ProPlot -- it imports your package and automatically figures out which classes + functions need to be documented, but when the class + function names change and you still have the auto-generated stub pages with the old names, you get errors.

I'll try adding a step in conf.py that empties out the api/*.rst stub pages generated by sphinx-automodapi every single time make html is run. This isn't the most efficient approach but should solve the problem.

Actually it's probably better to just "wipe" the build as described in #208 every time this happens. The main API changes quite rarely. Just triggered another build, hopefully it goes through.

[lukelbd]

Next we have a KeyError: PROJ_LIB issue associated with basemap as described in #5339. Basemap needs to use a custom environment variable, but since RTD does not use conda activate, this variable is unset. I'll copy their hack into my conf.py and try again.

[lukelbd]

Perfect, it's live! Thanks for the help. Luckily it seems the nbsphinx step doesn't need any more memory than the conda step.

[lukelbd]

@humitos I've been getting a lot of "Build exited due to time out" errors lately. The build is successful maybe ~10-20% of the time, seemingly randomly. I have to keep manually re-building. You'll notice a ton of failed builds in my build history.

Would it be possible to increase my maximum compile time?

[humitos]

I increased the time for your project and triggered a build at https://readthedocs.org/projects/proplot/builds/10567203/. Feel free to reopen if it does not work.