✨🚨Autoscaling: Prepare Warmed EBS-backed volumes to use as buffer for machines (⚠️ Devops) 🚨

[sanderegg]

What do these changes do?

This PR brings in the first step towards using stopped machines as a buffer instead of running machines. This PR brings in the background task that should create the buffer machines. The next PR will bring in using these.

NOTE: To use this new feature, it relies on new AMIs that always include a EBS volume, otherwise this cannot work! see https://git.speag.com/oSparc/osparc-infra/-/merge_requests/232

NOTE2: The new EBS-based buffering system is until now compatible with the old one.

Details

• driving test: services/autoscaling/tests/unit/test_modules_buffer_machine_core.py
• In a separate background task:
  1. Based on EC2_INSTANCES_ALLOWED_TYPES newly added field buffer_count,
  2. create buffer_count EC2 instances,
  3. via AWS SSM service, wait for EC2 to complete cloud-init by calling cloud-init --wait,
  4. via AWS SSM service, pre-pull images until completion,
  5. stop the machines,
  6. stopped machines are considered ready to use as buffer
• this is stateless, at each iteration the task checks for the current state directly via AWS EC2 API
• autoscaling (dynamic services only) newly need access to SSM acount: via AUTOSCALING_SSM_ACCESS
• autoscaling (dynamic services only) newly needs dedicated IAM role via EC2_INSTANCES_ATTACHED_IAM_PROFILE, such that it can create SSM enabled EC2 instances (AWS reference)

Bonus:

• now assert located in pytest-simcore helpers will nicely print reference

Next steps (next PR) shall be:

2nd step:

• if a dynamic service demands it,
• start a "stopped buffer machine" if available, changin its tags,
• use SSM to make the machine join the swarm,
• and let it live as usual...

possible improvements once all is in if necessary:

• handle changes in pre-pulled images - ✨Autoscaling: EBS-backed buffer, label EC2 machines with prepulled images list #6097
• use a cheap machine such as a t3 instead of using the real machine,
• possibly copy the volumes,
• possible recycle the volumes

Related issue/s

How to test

Dev-ops checklist

• new ENVs:

SSM_ACCESS_KEY_ID
SSM_ENDPOINT
SSM_REGION_NAME
SSM_SECRET_ACCESS_KEY
EC2_INSTANCES_ATTACHED_IAM_PROFILE

• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/675
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/640
• ⚠️ Please note that currently only AWS-master contains the necessary User + UserRoles! And therefore the ENV variables are empty!
• No ENV changes or I properly updated ENV (read the instruction)
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/640

[codecov]

Codecov Report

Attention: Patch coverage is 95.18900% with 14 lines in your changes missing coverage. Please review.

Project coverage is 87.9%. Comparing base (cafbf96) to head (268a547).
Report is 339 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #5923      +/-   ##
=========================================
+ Coverage    84.5%   87.9%    +3.3%     
=========================================
  Files          10    1437    +1427     
  Lines         214   59071   +58857     
  Branches       25    1404    +1379     
=========================================
+ Hits          181   51937   +51756     
- Misses         23    6840    +6817     
- Partials       10     294     +284     

Flag	Coverage Δ
integrationtests	64.8% <ø> (?)
unittests	85.9% <95.1%> (+1.4%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
packages/aws-library/src/aws_library/ec2/models.py	88.8% <100.0%> (ø)
...ling/src/simcore_service_autoscaling/api/health.py	100.0% <100.0%> (ø)
...rc/simcore_service_autoscaling/core/application.py	100.0% <100.0%> (ø)
...ing/src/simcore_service_autoscaling/core/errors.py	100.0% <100.0%> (ø)
...g/src/simcore_service_autoscaling/core/settings.py	100.0% <100.0%> (ø)
...oscaling/src/simcore_service_autoscaling/models.py	100.0% <100.0%> (ø)
...e_service_autoscaling/modules/auto_scaling_core.py	94.3% <100.0%> (ø)
...e_service_autoscaling/modules/auto_scaling_task.py	100.0% <100.0%> (ø)
...e_autoscaling/modules/buffer_machines_pool_task.py	100.0% <100.0%> (ø)
...ore_service_autoscaling/utils/auto_scaling_core.py	93.8% <100.0%> (ø)
... and 6 more

... and 1384 files with indirect coverage changes