Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🚀 Release v1.59.0 #4623

Closed
50 tasks done
mrnicegyu11 opened this issue Aug 17, 2023 · 7 comments
Closed
50 tasks done

🚀 Release v1.59.0 #4623

mrnicegyu11 opened this issue Aug 17, 2023 · 7 comments
Assignees
@mrnicegyu11 @pcrespov @YuryHrytsuk @matusdrobuliak66
Labels
release Preparation for pre-release/release t:maintenance Some planned maintenance work
Milestone
the nameless

Comments

@mrnicegyu11
Copy link
Member

mrnicegyu11 commented Aug 17, 2023 •
edited by YuryHrytsuk
Loading

Release version

1.59.0

Commit SHA

1d1c5d6b727b9fc4d7cd0c246e990c900498922c

Previous pre-release

https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_TheNameless2

Did the commit CI succeeded?

  • The commit CI succeeded.

Motivation

  • Regular release

Changes

Staging Sundae3

Release Issue: #4580

Staging Baklava1

Release Issue: n/a

Staging Baklava2

Release Issue: #4656

Staging Baklava3

Release Issue: #4700

Staging Baklava4

Release Issue: #4744

Staging Baklava5

Release Issue: #4755

Staging TheNameless1

Release Issue: #4772

Staging TheNameless2

Release Issue: #4784

Devops check 👷

(⚠️ DEVOPS) PRs

(⚠️ DEVOPS) Other planned changes

Tests assessment: e2e testing check 🧪

  • e2e tests on stagging looking standard nothing unusual (MD 8:30 27.9.)

Test assessment: targeted-testing 🔍️

No response

Test assessment: user-testing 🧐

No response

Summary 📝

  • Prepare release link
make release-prod version=1.59.0  git_sha=1d1c5d6b727b9fc4d7cd0c246e990c900498922c
  • Draft release changelog
  • Announce maintenance ( ** ANNOUNCE AT LEAST 24 HOURS BEFORE ** )
  • redis {"start": "2023-09-28T06:00:00.000Z", "end": "2023-09-28T10:00:00.000Z", "reason": "Release v1.59.0"}
    • aws
    • dalco
    • tip
  • status page (https://manage.statuspage.io/)
    • osparc
    • s4l
  • mattermost channels
    • maintenance
    • power users

Special Maintenance Tasks

URGENT

  • Update lite license server to remove unstructured Solvers @mguidon

Releasing 🚀

  • Maintenance page up.
cd /deployment/production/osparc-ops-environments
make up-maintenance
make down-maintenance
  • Check hanging sidecars. Helper command to run in director-v2 CLI simcore-service-director-v2 close-and-save-service <uuid>
  • Release by publishing draft
  • Check release CI
  • Check deployed
    • aws deploy
    • dalco deploy
    • tip deploy
  • Check testing endpoint ex. https://testing.osparc.speag.com/
  • Delete announcement
  • Check e2e runs
  • Announce
:tada: https://github.com/ITISFoundation/osparc-simcore/releases/tag/v<M.m.0>
@mrnicegyu11 mrnicegyu11 added t:maintenance Some planned maintenance work release Preparation for pre-release/release labels Aug 17, 2023
@mrnicegyu11
Copy link
Member Author

Hotfix in a very hacky way was deployed to tip.itis.swiss only. The docker image of webserver and static webserver was changed from release-github-latest to hotfix-github-latest

@mrnicegyu11 mrnicegyu11 reopened this Aug 17, 2023
@mrnicegyu11 mrnicegyu11 changed the title 🚀 Release v<M.m.0> 🚀 Release v1.59.0 Aug 17, 2023
@mrnicegyu11 mrnicegyu11 added this to the Baklava milestone Aug 17, 2023
@YuryHrytsuk
Copy link
Contributor

Be aware of removal of standardworker node label constraint. From now on, this label constraints should not appear in dy-sidecars and thus there is no need in having this label on docker nodes in cluster

@pcrespov pcrespov modified the milestones: Baklava, the nameless Sep 19, 2023
@YuryHrytsuk
Copy link
Contributor

When we release to prod, we need to rotate S3 Access Key and postgres_password.
We also need to restart deployment agent to make sure it uses new secrets (portainer password)

@matusdrobuliak66
Copy link
Contributor

@mrnicegyu11 @YuryHrytsuk Please check the error logs of traefik in the AWS Staging;

ime="2023-09-27T06:44:29Z" level=error msg="Router defined multiple times with different configurations in [monitoring-prometheuscadvisor-bs0zbli9yi7k3nkk4g7is7067 monitoring-prometheuscatchall-hfodm8skfcrsmgplruv2jbukp]" providerName=docker routerName=prometheus

Something is probably misconfigured, I expect the same problem in production with the release, so please make sure it is OK. thanks

@pcrespov
Copy link
Member

@mrnicegyu11 @YuryHrytsuk Please check the error logs of traefik in the AWS Staging;

ime="2023-09-27T06:44:29Z" level=error msg="Router defined multiple times with different configurations in [monitoring-prometheuscadvisor-bs0zbli9yi7k3nkk4g7is7067 monitoring-prometheuscatchall-hfodm8skfcrsmgplruv2jbukp]" providerName=docker routerName=prometheus

Something is probably misconfigured, I expect the same problem in production with the release, so please make sure it is OK. thanks

FYI: this is also reported in ITISFoundation/osparc-ops-environments#366

@pcrespov pcrespov mentioned this issue Sep 27, 2023
Closed
@matusdrobuliak66
Copy link
Contributor

matusdrobuliak66 commented Sep 28, 2023 •
edited
Loading

During this release when doing some manual configuration/fixes (because auto-deployment agent was not working properly @YuryHrytsuk @mrnicegyu11 please explain the issues more in detail) by mistake the master branch containers were deployed to production. This caused that the new commits with DB migration got in and therefore we decided we will fully release master.

What we did?

@matusdrobuliak66 matusdrobuliak66 mentioned this issue Sep 28, 2023
Closed
26 tasks
@mrnicegyu11
Copy link
Member Author

During the release we faces several issues:

  • The deployment agent 0.11.3 somehow mis-composed the env-vars which resulted in novel vars like RABBIT_SECURE missing. There will be follow-up talks on how to improve our handling of env-vars.
  • We switched to a manual generation of the docker-compose stack.yml file, and by human error we accidently deployed itisfoundation:master-github-latest docker images. This resulted in an unwanted, not reversible postgresql DB migration via alembic.
  • To mitigate the DB migration, we decided to release v1.60.0 as well, from the HEAD of osparc/simcore master. We deployed this using docker stack deploy with a manually generated docker-compose stack.yml file. We note that updating existing docker services lead to missing env-vars, but a full docker stack rm and subsequent docker stack deploy fixed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mrnicegyu11 mrnicegyu11

@pcrespov pcrespov

@YuryHrytsuk YuryHrytsuk

@matusdrobuliak66 matusdrobuliak66

Labels
release Preparation for pre-release/release t:maintenance Some planned maintenance work
Projects
None yet
Milestone
the nameless
Development

No branches or pull requests
4 participants
@mrnicegyu11 @pcrespov @YuryHrytsuk @matusdrobuliak66