Docker compose version missing in stac/config/magpie/ compose file

[mishaschwartz]

Overview

The version: key was not set in the stac/config/magpie/docker-compose-extra.yml file which caused docker-compose to report a version mismatch and fail to start.

Changes

Non-breaking changes

• Bug fix

Breaking changes
None

Related Issue / Discussion

Additional Information

Links to other issues or sources.

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

[tlvu]

This one is easy :D

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2084/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-compose-version-stac
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-20.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1325/

NOTEBOOK TEST RESULTS

    
[2023-09-21T20:13:28.939Z] ============================= test session starts ==============================
[2023-09-21T20:13:28.939Z] platform linux -- Python 3.9.16, pytest-7.3.1, pluggy-1.0.0
[2023-09-21T20:13:28.939Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-09-21T20:13:28.939Z] plugins: anyio-3.6.1, dash-2.10.0, nbval-0.9.6, tornasync-0.6.0.post2, xdist-3.3.1
[2023-09-21T20:13:28.939Z] collected 237 items
[2023-09-21T20:13:28.939Z] 
[2023-09-21T20:13:36.248Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-09-21T20:13:50.038Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2023-09-21T20:13:59.746Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2023-09-21T20:14:02.647Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb .FFFFFFF       [ 13%]
[2023-09-21T20:21:28.473Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 18%]
[2023-09-21T20:21:31.033Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 18%]
[2023-09-21T20:21:41.535Z] ...............                                                          [ 25%]
[2023-09-21T20:21:51.125Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-09-21T20:22:34.552Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 29%]
[2023-09-21T20:22:50.365Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-09-21T20:22:51.751Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-09-21T20:23:57.506Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-09-21T20:23:57.506Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-09-21T20:28:07.938Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 44%]
[2023-09-21T20:29:19.718Z] .............                                                            [ 49%]
[2023-09-21T20:29:24.168Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 51%]
[2023-09-21T20:29:26.398Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 54%]
[2023-09-21T20:29:41.774Z] .................                                                        [ 62%]
[2023-09-21T20:29:50.122Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 64%]
[2023-09-21T20:29:51.508Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 64%]
[2023-09-21T20:30:00.058Z] .........                                                                [ 68%]
[2023-09-21T20:30:10.626Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 73%]
[2023-09-21T20:30:19.363Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 75%]
[2023-09-21T20:30:21.271Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2023-09-21T20:30:24.076Z] ......                                                                   [ 78%]
[2023-09-21T20:30:30.698Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2023-09-21T20:30:46.196Z] .............                                                            [ 84%]
[2023-09-21T20:30:56.209Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2023-09-21T20:31:30.332Z] ....s.                                                                   [ 87%]
[2023-09-21T20:31:38.446Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2023-09-21T20:31:50.983Z] ...                                                                      [ 89%]
[2023-09-21T20:32:05.895Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2023-09-21T20:32:27.882Z] ......                                                                   [ 92%]
[2023-09-21T20:32:28.978Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2023-09-21T20:35:03.130Z] notebooks/stress-tests.ipynb ......                                      [100%]
[2023-09-21T20:35:03.130Z] 
[2023-09-21T20:35:03.130Z] =================================== FAILURES ===================================
    
  

[fmigneault]

This is something I intentionally left out, because as soon as any version mismatches anywhere, combining them fails. So it is somewhat redundant to have it duplicated in each file. We could have it defined only once in https://github.com/bird-house/birdhouse-deploy/blob/master/birdhouse/docker-compose.yml.

Not against adding it explicitly, though. Just wanted to share why it was missing.

[tlvu]

Not against adding it explicitly, though. Just wanted to share why it was missing.

I agree the version key is optional. We should remove them all. However, when doing this, all external repos should be done at the same time. And I think we should do it eventually but communications has to very clear with all partners because this is a breaking change.

For now, if we do not include the version key, the entire stack will not start and we have to ensure all changes are not breaking change, unless it is scheduled and communicated very clearly in advance.

[mishaschwartz]

@fmigneault

This is something I intentionally left out

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

I'm very very happy to update the docker compose version though instead

[tlvu]

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

The physical host has the same docker-compose version as what vagrant installs, which is also the same version as the autodeploy container.

birdhouse-deploy/birdhouse/vagrant-utils/install-docker.sh

Lines 102 to 105 in 2045cc5

	# Pin docker-compose to a specific version to match autodeploy version of
	# docker-compose. Otherwise, alternating between the different versions will
	# keep recreating all the containers for no reasons.
	LATEST_COMPOSE_VERSION="1.26.2"

The reason to keep all the versions in-sync is documented in the comment above. When there is a mismatch, the containers will re-create themselves all the time, even for a simple ./pavics-compose.sh up -d that should do nothing because no change to any configs.

I'm very very happy to update the docker compose version though instead

Updating the docker-compose version means all the physical hosts of all existing deployments have to do it at the same time.

I was thinking of a solution less disruptive documented here #246

All comments about docker-compose update should probably stay in that issue #246 to keep everything organized.

[fmigneault]

@mishaschwartz

Which version of docker compose are you using though? Because older ones (like the version suggested in the vagrant installation scripts) will complain about the mismatch.

I'm currently running v2.17.2, but was running 1.2x before. Both complained the same that it could not merge YAMLs with
different version values.

[fmigneault]

I've tested on a older instance using docker-compose v1.19 and I do see an invalid "version 1" being used by default due to the missing version. It doesn't break the instance startup, but we can avoid the unnecessary error by adding it explicitly.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/2098/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : fix-compose-version-stac
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-166.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov