🎨Clusters keeper/use ssm (🚨change in private clusters)

[sanderegg]

What do these changes do?

⚠️ This should go in master AFTER the release

Currently when private clusters are created, a new EC2 is created called the primary or manager instance. The current workflow is as follows:

• EC2 launch,
• EC2 configuration through custom boot scripts, (through custom bash script on EC2 launch)
• pulling of dask TLS data from AWS SSM, (through custom bash script on EC2 launch)
• docker swarm initialization, (through custom bash script on EC2 launch)
• docker stack deployment with all necessary ENV setup (through custom bash script on EC2 launch),

All these custom bash scripts are passed to the EC2 as so-called UserData (special text that is run on start). The UserData is limited to 16 KB and we are hitting this limit.

This PR changes this behaviour by splitting the part in UserData and the part run via AWS SSM service:
AFTER:

• EC2 launch,
• EC2 configuration through custom boot scripts, (through custom bash script on EC2 launch)
• pulling of dask TLS data from AWS SSM, (via SSM command)
• docker swarm initialization, (via SSM command)
• docker stack deployment with all necessary ENV setup (via SSM command),

In order to make this work, the clusters-keeper now waits for the EC2 to connect to the SSM server, and then send the SSM command. This might slightly reduce performances but is expected to be not noticeable.

⚠️ SSM Access is now necessary for clusters-keeper as well

NOTE:

• SSM command size is limited to 64KB,
• in case it becomes too large, the commands might be split in several commands.

NOTE2: This will allow to correctly setup graylog in computational clusters,
NOTE3: This will allow to correctly setup registry mirroring in computational clusters,
NOTE4: This will allow to add more machine types without breaking the limit,

Related issue/s

• fixes Clusters-keeper: cannot cope with long custom scripts #6249

How to test

Dev-ops checklist

new ENV:

• PRIMARY_EC2_INSTANCES_MAX_START_TIME: The env was missing from docker-compose file
• PRIMARY_EC2_INSTANCES_DOCKER_DEFAULT_ADDRESS_POOL: default address pool used by the private clusters swarms
• CLUSTERS_KEEPER_SSM_ACCESS: the SSM access credentials for Clusters-keeper @YuryHrytsuk we need a special clusters-keeper SSM user
• ENV changes MR: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/900
• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 98.98990% with 1 line in your changes missing coverage. Please review.

Project coverage is 88.2%. Comparing base (cafbf96) to head (8d03db6).
Report is 563 commits behind head on master.

Files with missing lines	Patch %	Lines
.../simcore_service_clusters_keeper/utils/clusters.py	87.5%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6361      +/-   ##
=========================================
+ Coverage    84.5%   88.2%    +3.6%     
=========================================
  Files          10    1507    +1497     
  Lines         214   62485   +62271     
  Branches       25    2070    +2045     
=========================================
+ Hits          181   55140   +54959     
- Misses         23    7025    +7002     
- Partials       10     320     +310     

Flag	Coverage Δ
integrationtests	64.6% <ø> (?)
unittests	86.2% <98.9%> (+1.6%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ages/models-library/src/models_library/clusters.py	100.0% <100.0%> (ø)
.../src/simcore_service_clusters_keeper/api/health.py	100.0% <100.0%> (ø)
...r/src/simcore_service_clusters_keeper/constants.py	100.0% <100.0%> (ø)
...imcore_service_clusters_keeper/core/application.py	100.0% <100.0%> (ø)
...c/simcore_service_clusters_keeper/core/settings.py	100.0% <100.0%> (ø)
...imcore_service_clusters_keeper/modules/clusters.py	100.0% <ø> (ø)
...lusters_keeper/modules/clusters_management_core.py	100.0% <100.0%> (ø)
...lusters_keeper/modules/clusters_management_task.py	100.0% <ø> (ø)
...src/simcore_service_clusters_keeper/modules/ssm.py	100.0% <100.0%> (ø)
...r/src/simcore_service_clusters_keeper/utils/ec2.py	100.0% <100.0%> (ø)
... and 1 more

... and 1446 files with indirect coverage changes

[mrnicegyu11]

read the description not the code, ask if I shall read through it (serious offer).

Also, kudos on going from the 16k limit to the 64k limit 🍡

[sanderegg]

Blocked until release is done

[pcrespov]

thx!

[sonarqubecloud]

Quality Gate failed

Failed conditions
19.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud