New Jupyter env build with new xlrd, pre-commit, pin dask, distributed, cf_xarray, latest of everything else #101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…problem dask and distributed for slow PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb. cf_xarray for slow WPS_example.ipynb.
…blems Same as in commit bird-house/finch@7bf4a7e
As requested by Jeremy Fyke.
… /opt/conda/envs/birdy/fonts" The warning appears only once and may or may not re-appear again after a "Restart kernel". It is completely intermittent. See issue #97.
…ance problems" This reverts commit 1387921. Problem seems to be done by itself.
…ests:220401.1" This reverts commit 406ad0e. The performance problem is still there if we do not pin dask, distributed and cf_xarray.
… performance problems"" This reverts commit 9447d53. Testing error, the performance problem is still there if we do not keep those pins.
This was referenced Apr 1, 2022
tlvu
changed the title
tlogan2000
approved these changes
Apr 4, 2022
|
@tlogan2000 this is deployed as "beta" image in production, could you do a quick bokeh visualization performance test to spot for regressions that we've seen in the past? |
tlvu
added a commit
to bird-house/birdhouse-deploy
that referenced
this pull request
Apr 7, 2022
jupyter env: new xlrd, pre-commit, pin dask, distributed, cf_xarray, latest of everything else Deploy new Jupyter env from PR Ouranosinc/PAVICS-e2e-workflow-tests#101 on PAVICS. Detailed changes can be found at Ouranosinc/PAVICS-e2e-workflow-tests#101. Relevant changes: ```diff > - pre-commit=2.17.0=py38h578d9bd_0 > - xlrd=2.0.1=pyhd8ed1ab_3 < - xclim=0.32.1=pyhd8ed1ab_0 > - xclim=0.34.0=pyhd8ed1ab_0 < - cfgrib=0.9.9.1=pyhd8ed1ab_1 > - cfgrib=0.9.10.1=pyhd8ed1ab_0 < - cftime=1.5.1.1=py38h6c62de6_1 > - cftime=1.6.0=py38h3ec907f_0 < - intake-xarray=0.5.0=pyhd8ed1ab_0 > - intake-xarray=0.6.0=pyhd8ed1ab_0 < - pandas=1.3.5=py38h43a58ef_0 > - pandas=1.4.1=py38h43a58ef_0 < - regionmask=0.8.0=pyhd8ed1ab_1 > - regionmask=0.9.0=pyhd8ed1ab_0 < - rioxarray=0.9.1=pyhd8ed1ab_0 > - rioxarray=0.10.3=pyhd8ed1ab_0 < - xarray=0.20.2=pyhd8ed1ab_0 > - xarray=2022.3.0=pyhd8ed1ab_0 < - zarr=2.10.3=pyhd8ed1ab_0 > - zarr=2.11.1=pyhd8ed1ab_0 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant Changes
New Packages Pins
daskanddistributedwas causingPAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynbto run excessively slow.cf_xarraycausedclimex.ipynbto be slow.See #100 for more info why these had to be pinned.
Related Issue / Discussion
Test Results
Deployed as "beta" image in production for bokeh visualization performance regression testing.
Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine.
Jenkins build, only known intermittent error: subset-user-input.ipynb: RemoteDisconnected, slow WPS_example.ipynb: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/preview-docker-build/37/console, logs: job-PAVICS-e2e-workflow-tests-preview-docker-build-37-consoleText.txt
Jenkins build 2, only known intermittent error: subset-user-input.ipynb: RemoteDisconnected, opendap.ipynb runtime crashed: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build/7/console, logs: job-PAVICS-e2e-workflow-tests-new-docker-build-7-consoleText.txt
Jenkins build 3, only known intermittent error: stress-test.ipynb "ValueError: max() arg is an empty sequence": http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build/8/console, logs: job-PAVICS-e2e-workflow-tests-new-docker-build-8-consoleText.txt
Jenkins build 4, only known intermittent error: WMS_example.ipynb: SyntaxError: not a PNG file, stress-tests.ipynb: ValueError: max() arg is an empty sequence, , slow WPS_example.ipynb: http://jenkins.ouranos.ca/job/PAVICS-e2e-workflow-tests/job/new-docker-build/12/console, logs: job-PAVICS-e2e-workflow-tests-new-docker-build-12-consoleText.txt
Additional Information
Full
conda env exportdiff:220121-220401-conda-env-export.diff.txt
Full new
conda env export:220401-conda-env-export.yml.txt