docker: prevent manual pin of dependencies and improve build speed #95

tlvu · 2021-12-21T21:46:41Z

Overview

Previously, when xclim and ravenpy were pinning their own dependencies, the pins were ignored and we had to manually repeat the same pins again. See comment #94 (comment).

This PR allows xclim and ravenpy to manage their own dependencies pinning transparently during this Jupyter env rebuild.

Also fixed a long standing build performance along the way. Build time went from 50 mins to 25 mins and builds on DockerHub works again (fixes #51).

Deployed as "beta" image on https://pavics.ouranos.ca/jupyter for testing.

Changes

Switched to using mamba instead of conda since mamba dependency solver is faster. Mamba solver being faster at the expense of less precision so had to pin latest xclim and ravenpy to avoid random downgrade in the 2nd build phase.

Both solvers performance seem to drop exponentially when less packages are specified directly, leading to more work for the solver to discover them. Less packages specified directly because we removed all direct dependencies of xclim and ravenpy from the environment.yml file.
Switched to using 2 stages conda env build as another performance work-around. One single conda env create -f /environment.yml was taking many days ! mamba was not much better in one single stage build.

With 2 stages build, a build using conda solver takes 5 hours while mamba solver takes 25 minutes !
Reduced the number of "build layers" by merging several of them, for another small build performance gain.
jupyterlab-topbar-text and jupyterlab-theme-toggle jupyterlab extension was removed due to javascript build problem. The topbar text was pretty useless. Hopefully the theme toogle is not so widely used.
Had to hardcode the commit of the https://github.com/jupyter/docker-stacks repo where we get the startup script from because the latest version of those scripts are breaking us. This will have to be solve later.
Removed vcs library from cdat channel in order to move to python 3.9. Otherwise we are stuck on 3.7 and xarray will drop 3.7 soon. I've opened an issue on CDAT side Unable to install vcs conda package in a python 3.9 environment: package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed CDAT/vcs#457. vcs library was needed to run ESGF notebooks at https://github.com/ESGF/esgf-compute-api/tree/devel/examples

Related Issue / Discussion

Related issues Unable to run 'jupyter lab build' with jupyterlab 3.2.5, previous 3.2.4 was working fine jupyterlab/jupyterlab#11726
Notebook fix needed fix notebook 5 - take 2 PAVICS-landing#42
Matching PR to deploy this new Jupyter env to PAVICS Jupyter: update Jupyter env for latest XClim, RavenPy and all packages bird-house/birdhouse-deploy#234

Additional Information

Screenshot of UI change showing jupyterlab-topbar-text and jupyterlab-theme-toggle jupyterlab extension removed:
Relevant changes:

<   - xclim=0.31.0=pyhd8ed1ab_0
>   - xclim=0.32.1=pyhd8ed1ab_0

<   - ravenpy=0.7.5=pyhff6ddc9_0
>   - ravenpy=0.7.8=pyh8a188c0_0

<   - python=3.7.12=hb7a2778_100_cpython
>   - python=3.8.12=hb7a2778_2_cpython

# removed
<   - vcs=8.2.1=pyh9f0ad1d_0

<   - numpy=1.21.4=py37h31617e3_0
>   - numpy=1.21.5=py38h87f13fb_0

<   - xarray=0.20.1=pyhd8ed1ab_0
>   - xarray=0.20.2=pyhd8ed1ab_0

<   - rioxarray=0.8.0=pyhd8ed1ab_0
>   - rioxarray=0.9.1=pyhd8ed1ab_0

<   - cf_xarray=0.6.1=pyh6c4a22f_0
>   - cf_xarray=0.6.3=pyhd8ed1ab_0

<   - gdal=3.3.2=py37hd5a0ba4_2
>   - gdal=3.3.3=py38hcf2042a_0

<   - rasterio=1.2.6=py37hc20819c_2
>   - rasterio=1.2.10=py38hfd64e68_0

<   - climpred=2.1.6=pyhd8ed1ab_1
>   - climpred=2.2.0=pyhd8ed1ab_0

<   - clisops=0.7.0=pyh6c4a22f_0
>   - clisops=0.8.0=pyh6c4a22f_0

<   - xesmf=0.6.0=pyhd8ed1ab_0
>   - xesmf=0.6.2=pyhd8ed1ab_0

<   - birdy=v0.8.0=pyh6c4a22f_1
>   - birdy=0.8.1=pyh6c4a22f_1

<   - cartopy=0.20.0=py37hbe109c4_0
>   - cartopy=0.20.1=py38hf9a4893_1

<   - dask=2021.11.2=pyhd8ed1ab_0
>   - dask=2022.1.0=pyhd8ed1ab_0

<   - numba=0.53.1=py37hb11d6e1_1
>   - numba=0.55.0=py38h4bf6c61_0

<   - pandas=1.3.4=py37he8f5f7f_1
>   - pandas=1.3.5=py38h43a58ef_0

Full diff of conda env export:

211123-update211216-211221-conda-env-export.diff.txt

211221-220116.1-conda-env-export.diff.txt

211123-update211216-220116.1-conda-env-export.diff.txt

220116.1-220121-conda-env-export.diff.txt

211123-update211216-220121-conda-env-export.diff.txt

Full new conda env export:

211221-conda-env-export.yml.txt

220116.1-conda-env-export.yml.txt

220121-conda-env-export.yml.txt

…yter lab v3 anymore, causing us build problems now

…ance

…e solver

…avoid too big dependency resolution

…ot found

Encountered problems while solving: - package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed

This reverts commit 498bc8c. vtk-cdat was already part of cdat repo.

…estigate later

…mba update

… if it prevent the downgrade

…try to pin in environment.yml as well

…ate hardcode pin in Dockerfile

…enssl significantly ``` - # - openssl 3.0.0 h7f98852_2 installed - # + openssl 1.1.1l h7f98852_0 conda-forge/linux-64 2 MB ```

tlvu · 2022-01-14T18:35:30Z

FYI @ChaamC I am at the end of road with this jupyter-conda plugin problem, I've opened an issue on their side to get more help mamba-org/gator#170

docker/environment.yml

See mamba-org/gator#170.

Not sure why cartopy-0.14.3 is selected! Previous build selected cartopy-0.20.1 and geos-3.9.1 was found! ``` Encountered problems while solving: - nothing provides geos 3.5.* needed by cartopy-0.14.3-np110py27_4 ```

Zeitsperre

Python3.10 makes breaking changes to several standard libraries (collections, types, typing). It looks as though we might need to make changes to a few dependencies (or help push new versions) if we want to maintain support for newer versions.

Zeitsperre · 2022-01-17T16:32:40Z

docker/Dockerfile

+# Conda was stuck at this step:
+# DEBUG conda.common._logic:_run_sat(607): Invoking SAT with clause count: 2500273
+#
+# Pin python=3.9 because python 3.10 cause this error:


Additionally, PyDAP is not Python3.10 compatible. On import:

... File ~/miniconda3/envs/pavics-sdi/lib/python3.10/site-packages/pydap/model.py:175, in <module> 173 from six import string_types 174 import numpy as np --> 175 from collections import OrderedDict, Mapping 176 import warnings 178 from .lib import quote, decode_np_strings ImportError: cannot import name 'Mapping' from 'collections' (/home/me/miniconda3/envs/pavics-sdi/lib/python3.10/collections/__init__.py)

tlvu · 2022-01-18T00:01:21Z

@tlogan2000

I pinned shapely to the old 1.7.1 version and there is a new failure with the homepage nb 5: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/prevent-manual-pin-of-dependencies/14/consoleFull

12:36:36  File /opt/conda/envs/birdy/lib/python3.9/site-packages/holoviews/core/data/xarray.py:224, in XArrayInterface.init(cls, eltype, data, kdims, vdims)
12:36:36      220     undeclared = [
12:36:36      221         c for c in da.coords if c not in kdims and len(da[c].shape) == 1 and
12:36:36      222         da[c].shape[0] > 1]
12:36:36      223     if undeclared:
12:36:36  --> 224         raise DataError(
12:36:36      225             'The coordinates on the %r DataArray do not match the '
12:36:36      226             'provided key dimensions (kdims). The following coords '
12:36:36      227             'were left unspecified: %r. If you are requesting a '
12:36:36      228             'lower dimensional view such as a histogram cast '
12:36:36      229             'the xarray to a columnar format using the .to_dataframe '
12:36:36      230             'or .to_dask_dataframe methods before providing it to '
12:36:36      231             'HoloViews.' % (vdim.name, undeclared))
12:36:36      232 return data, {'kdims': kdims, 'vdims': vdims}, {}
12:36:36  
12:36:36  DataError: The coordinates on the 'tx_mean' DataArray do not match the provided key dimensions (kdims). The following coords were left unspecified: ['horizon']. If you are requesting a lower dimensional view such as a histogram cast the xarray to a columnar format using the .to_dataframe or .to_dask_dataframe methods before providing it to HoloViews.

A good news is the bokeh/holoviews performance problem seems to be fixed in the new build.

This new build is deployed as "beta" image in prod.

tlogan2000 · 2022-01-18T12:58:40Z

I'll take a look.

Sample error: ``` _______ pavics-sdi-master/docs/source/notebooks/forecasts.ipynb::Cell 2 ________ Notebook cell execution failed Cell 2: Cell outputs differ Input: import hvplot.xarray mtl = ds.sel(lon=45.5, lat=360 - 73.5, method="nearest") mtl.pr.hvplot(x="time", width=300) + mtl.tas.hvplot(x="time", width=300) Traceback: mismatch 'application/vnd.holoviews_load.v0+json' assert reference_output == test_output failed: '\n(function(...\n}(window));' == '\n(function(...\n}(window));' Skipping 1273 identical leading characters in diff, use -v to show , now()); + root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length; function on_load() { root._bokeh_is_loading--; if (root._bokeh_is_loading === 0) { console.debug("Bokeh: all BokehJS libraries/stylesheets loaded"); run_callbacks() } } ( ... ) ```

tlvu · 2022-01-20T23:50:21Z

@ChaamC FYI the jupyter-conda plugin is fixed by uninstalling mamba from the image !

Deploy new Jupyter env from PR Ouranosinc/PAVICS-e2e-workflow-tests#95 on PAVICS. Detailed changes can be found at Ouranosinc/PAVICS-e2e-workflow-tests#95. Relevant changes: ```diff < - xclim=0.31.0=pyhd8ed1ab_0 > - xclim=0.32.1=pyhd8ed1ab_0 < - ravenpy=0.7.5=pyhff6ddc9_0 > - ravenpy=0.7.8=pyh8a188c0_0 < - python=3.7.12=hb7a2778_100_cpython > - python=3.8.12=hb7a2778_2_cpython < - vcs=8.2.1=pyh9f0ad1d_0 < - numpy=1.21.4=py37h31617e3_0 > - numpy=1.21.5=py38h87f13fb_0 < - xarray=0.20.1=pyhd8ed1ab_0 > - xarray=0.20.2=pyhd8ed1ab_0 < - rioxarray=0.8.0=pyhd8ed1ab_0 > - rioxarray=0.9.1=pyhd8ed1ab_0 < - cf_xarray=0.6.1=pyh6c4a22f_0 > - cf_xarray=0.6.3=pyhd8ed1ab_0 < - gdal=3.3.2=py37hd5a0ba4_2 > - gdal=3.3.3=py38hcf2042a_0 < - rasterio=1.2.6=py37hc20819c_2 > - rasterio=1.2.10=py38hfd64e68_0 < - climpred=2.1.6=pyhd8ed1ab_1 > - climpred=2.2.0=pyhd8ed1ab_0 < - clisops=0.7.0=pyh6c4a22f_0 > - clisops=0.8.0=pyh6c4a22f_0 < - xesmf=0.6.0=pyhd8ed1ab_0 > - xesmf=0.6.2=pyhd8ed1ab_0 < - birdy=v0.8.0=pyh6c4a22f_1 > - birdy=0.8.1=pyh6c4a22f_1 < - cartopy=0.20.0=py37hbe109c4_0 > - cartopy=0.20.1=py38hf9a4893_1 < - dask=2021.11.2=pyhd8ed1ab_0 > - dask=2022.1.0=pyhd8ed1ab_0 < - numba=0.53.1=py37hb11d6e1_1 > - numba=0.55.0=py38h4bf6c61_0 < - pandas=1.3.4=py37he8f5f7f_1 > - pandas=1.3.5=py38h43a58ef_0 ```

Jupyter: update Jupyter env for latest XClim, RavenPy and all packages Deploy new Jupyter env from PR Ouranosinc/PAVICS-e2e-workflow-tests#95 on PAVICS. Detailed changes can be found at Ouranosinc/PAVICS-e2e-workflow-tests#95. Relevant changes: ```diff < - xclim=0.31.0=pyhd8ed1ab_0 > - xclim=0.32.1=pyhd8ed1ab_0 < - ravenpy=0.7.5=pyhff6ddc9_0 > - ravenpy=0.7.8=pyh8a188c0_0 < - python=3.7.12=hb7a2778_100_cpython > - python=3.8.12=hb7a2778_2_cpython < - vcs=8.2.1=pyh9f0ad1d_0 < - numpy=1.21.4=py37h31617e3_0 > - numpy=1.21.5=py38h87f13fb_0 < - xarray=0.20.1=pyhd8ed1ab_0 > - xarray=0.20.2=pyhd8ed1ab_0 < - rioxarray=0.8.0=pyhd8ed1ab_0 > - rioxarray=0.9.1=pyhd8ed1ab_0 < - cf_xarray=0.6.1=pyh6c4a22f_0 > - cf_xarray=0.6.3=pyhd8ed1ab_0 < - gdal=3.3.2=py37hd5a0ba4_2 > - gdal=3.3.3=py38hcf2042a_0 < - rasterio=1.2.6=py37hc20819c_2 > - rasterio=1.2.10=py38hfd64e68_0 < - climpred=2.1.6=pyhd8ed1ab_1 > - climpred=2.2.0=pyhd8ed1ab_0 < - clisops=0.7.0=pyh6c4a22f_0 > - clisops=0.8.0=pyh6c4a22f_0 < - xesmf=0.6.0=pyhd8ed1ab_0 > - xesmf=0.6.2=pyhd8ed1ab_0 < - birdy=v0.8.0=pyh6c4a22f_1 > - birdy=0.8.1=pyh6c4a22f_1 < - cartopy=0.20.0=py37hbe109c4_0 > - cartopy=0.20.1=py38hf9a4893_1 < - dask=2021.11.2=pyhd8ed1ab_0 > - dask=2022.1.0=pyhd8ed1ab_0 < - numba=0.53.1=py37hb11d6e1_1 > - numba=0.55.0=py38h4bf6c61_0 < - pandas=1.3.4=py37he8f5f7f_1 > - pandas=1.3.5=py38h43a58ef_0 ```

tlvu added 30 commits December 17, 2021 17:28

docker: try mamba

d21f8e2

docker: enable DEBUG logging for conda

e7a9640

docker: remove direct dependencies from xclim and ravenpy

e1e9fba

docker: remove 'jupyter lab build' step, probably not needed with jup…

2ef396c

…yter lab v3 anymore, causing us build problems now

docker: try set channel_priority strict to hopefully increase perform…

f71e61e

…ance

docker: try to pin xclim and ravenpy to hopefully increase performance

3a5472f

docker: use --strict-channel-priority with conda env create to be sure

6f8f0ea

docker: remove defaults channel to decrease number of packages for th…

53e81d7

…e solver

docker: replace conda env create with conda create

7bb511c

docker: replace conda create needs --name

335b4b2

docker: use --file instead -f for conda create

f8df4cf

docker: conda create do not work, revert to conda env create

23b8912

docker: install mamba the proper way and use mamba env create

f992cef

docker: unpin xclim and ravenpy to try with mamba

965860b

docker: use mamba to create conda env and perform 2-stage install to …

eafacd9

…avoid too big dependency resolution

docker: need to specify channels for manual mamba create

0452d4c

docker: channel conda-forge before defaults, gdal needed by ravenpy n…

26fa54f

…ot found

docker: typo: mamba env install and not just mamba install

ff696db

docker: typo: mamba env update and not mamba env install

bf768d9

docker: add vtk-cdat channel for vtk-cdat, needed by vcs

498bc8c

Encountered problems while solving: - package vcs-8.1-py_0 requires vtk-cdat >8.1, but none of the providers can be installed

Revert "docker: add vtk-cdat channel for vtk-cdat, needed by vcs"

6d13a42

This reverts commit 498bc8c. vtk-cdat was already part of cdat repo.

docker: remove vcs for weird dependency solve error, will have to inv…

3e8cfc5

…estigate later

docker: temp change to leverage build caching

9860b0d

docker: optimize for cache re-use during dev rebuild cycle

e0f5325

docker: try hack to ensure latest xclim and ravenpy

f105f91

docker: mamba install do nothing if already installed, have to use ma…

36a4b0f

…mba update

docker: mamba update did not update either, force install exact version

f5fa5d7

docker: try pin xclim and ravenpy immediately at the beginning to see…

66953c0

… if it prevent the downgrade

docker: even with pin in initial install, still getting downgrade so …

87350e8

…try to pin in environment.yml as well

docker: pin in environment.yml stopped the downgrading, remove duplic…

cc2d7ef

…ate hardcode pin in Dockerfile

docker: revert change to build as regular user since it downgrades op…

8aba3a5

…enssl significantly ``` - # - openssl 3.0.0 h7f98852_2 installed - # + openssl 1.1.1l h7f98852_0 conda-forge/linux-64 2 MB ```

Zeitsperre reviewed Jan 14, 2022

View reviewed changes

docker/environment.yml Outdated Show resolved Hide resolved

docker: xesmf-0.6.2 requires clisops>=0.8.0, new ravenpy available

c9d2ef6

tlvu mentioned this pull request Jan 14, 2022

Update xESMF #96

Closed

tlvu added 6 commits January 14, 2022 14:38

docker: remove mamba for mamba_gator plugin to work properly

107c402

See mamba-org/gator#170.

docker: pin shapely due to notebook failure

8bd0094

release: update to use image pavics/workflow-tests:220116

c90413c

docker: prevent accidental cartopy downgrade from mamba

9370388

Not sure why cartopy-0.14.3 is selected! Previous build selected cartopy-0.20.1 and geos-3.9.1 was found! ``` Encountered problems while solving: - nothing provides geos 3.5.* needed by cartopy-0.14.3-np110py27_4 ```

docker: pin python=3.9

58cd714

release: update to use image pavics/workflow-tests:220116.1

d58f090

Zeitsperre reviewed Jan 17, 2022

View reviewed changes

huard mentioned this pull request Jan 17, 2022

Notebook showcasing the ECCC geoapi Ouranosinc/pavics-sdi#245

Merged

tlvu added 2 commits January 21, 2022 12:30

docker: pin python=3.8 for xESMF

007ae84

release: update to use image pavics/workflow-tests:220121

34d38cc

This was referenced Jan 22, 2022

Drop Python3.7 Ouranosinc/xclim#966

Closed

fix notebook 5 - take 2 Ouranosinc/PAVICS-landing#42

Merged

tlvu mentioned this pull request Jan 27, 2022

Jupyter: update Jupyter env for latest XClim, RavenPy and all packages bird-house/birdhouse-deploy#234

Merged

tlvu merged commit 607ec58 into master Jan 27, 2022

tlvu deleted the prevent-manual-pin-of-dependencies branch January 27, 2022 14:36

tlvu mentioned this pull request Feb 15, 2022

DockerHub build failed at conda env create command with connection broken error #51

Closed

This was referenced Mar 18, 2022

Re-enable the ESGF notebooks for the CWT API #13

Closed

Need to unpin shapely in Jupyter env #99

Closed

tlvu mentioned this pull request Jul 4, 2022

Tutorial notebooks overhaul Ouranosinc/raven#424

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker: prevent manual pin of dependencies and improve build speed #95

docker: prevent manual pin of dependencies and improve build speed #95

tlvu commented Dec 21, 2021 •

edited

Loading

tlvu commented Jan 14, 2022

Zeitsperre left a comment

Zeitsperre Jan 17, 2022

tlvu commented Jan 18, 2022

tlogan2000 commented Jan 18, 2022

tlvu commented Jan 20, 2022

docker: prevent manual pin of dependencies and improve build speed #95

docker: prevent manual pin of dependencies and improve build speed #95

Conversation

tlvu commented Dec 21, 2021 • edited Loading

Overview

Changes

Related Issue / Discussion

Additional Information

tlvu commented Jan 14, 2022

Zeitsperre left a comment

Choose a reason for hiding this comment

Zeitsperre Jan 17, 2022

Choose a reason for hiding this comment

tlvu commented Jan 18, 2022

tlogan2000 commented Jan 18, 2022

tlvu commented Jan 20, 2022

tlvu commented Dec 21, 2021 •

edited

Loading