✨Autoscaling: 1st draft on auto-scaling computational clusters

[sanderegg]

What do these changes do?

This allows the autoscaling service to optionally connect to a dask-scheduler to find out whether it is in need of dask-workers to complete pending jobs.

For context:

• a user starts a job (though publicAPI, or through the osparc UI) that should run on on-demand clusters
• the clusters-keeper service creates a primary machine on AWS EC2 where a dask-scheduler and autoscaling shall run [upcoming PRs],
• the autoscaling service connects to the dask-scheduler and monitors for unrunnable jobs, if it finds some it will try to create a machine that accomodates the job required resources, [THIS PR]
• the newly created machine will auto-start a dask-sidecar, that auto-connects to the dask-scheduler as soon as the machine is available, [THIS PR]
• the job runs, and the results are fetched by the director-v2
• the autoscaling then removes the machine if it is unused for some time, [THIS PR]

details

• adds dask distributed dependencies
• removes old outdated script.py from sandbox
• adds computational banner on start in the autoscaling logs
• new ENV DASK_MONITORING_URL for use when autoscaling is to be set in computational mode (NOTE: DASK_MONITORING_URL and NODE_MONITORING settings cannot be both set)
• new ENV EC2_INSTANCES_NAME_PREFIX defaults to "autoscaling" allows to prefix machines created by the autoscaling service

Related issue/s

• requires ♻️Autoscaling: prepare for computational autoscaling #4709
• requires ♻️ Autoscaling: refactor before 1st draft of computational autoscaling (⚠️ devops) #4874

How to test

make test-dev-unit

otherwise manual testing see tests/manual/README.md

DevOps Checklist

• new optional ENV EC2_INSTANCES_NAME_PREFIX, which default to "autoscaling" to keep backwards compatibility with dynamic services autoscaling.

[codecov]

Codecov Report

Merging #4711 (b149147) into master (ff62322) will increase coverage by 0.2%.
The diff coverage is 98.8%.

@@           Coverage Diff            @@
##           master   #4711     +/-   ##
========================================
+ Coverage    86.6%   86.8%   +0.2%     
========================================
  Files        1188    1191      +3     
  Lines       49402   49656    +254     
  Branches     1072    1072             
========================================
+ Hits        42825   43145    +320     
+ Misses       6342    6276     -66     
  Partials      235     235             

Flag	Coverage Δ
integrationtests	65.1% <ø> (+1.3%)	⬆️
unittests	84.5% <98.8%> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...toscaling/src/simcore_service_autoscaling/_meta.py	100.0% <100.0%> (ø)
...rc/simcore_service_autoscaling/core/application.py	100.0% <100.0%> (ø)
...ing/src/simcore_service_autoscaling/core/errors.py	100.0% <100.0%> (ø)
...g/src/simcore_service_autoscaling/core/settings.py	100.0% <100.0%> (ø)
...oscaling/src/simcore_service_autoscaling/models.py	100.0% <100.0%> (ø)
...e_service_autoscaling/modules/auto_scaling_task.py	100.0% <100.0%> (ø)
...ore_service_autoscaling/utils/auto_scaling_core.py	90.4% <100.0%> (-9.6%)	⬇️
...service_autoscaling/utils/computational_scaling.py	100.0% <100.0%> (ø)
...aling/src/simcore_service_autoscaling/utils/ec2.py	100.0% <100.0%> (ø)
.../src/simcore_service_autoscaling/utils/rabbitmq.py	100.0% <100.0%> (ø)
... and 3 more

... and 12 files with indirect coverage changes

[GitHK]

Looks good from what I can tell

[pcrespov]

Great job! Just left some comments and suggestions. I am curious about a few things

[codeclimate]

Code Climate has analyzed commit b149147 and detected 0 issues on this pull request.

View more on Code Climate.

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
4.1% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint