✨Autoscaling: add buffer metrics

[sanderegg]

What do these changes do?

This adds metrics for the buffer pooling:

• in essence in the next days, every type of EC2 allowed to auto-scale will/can have a different number of buffer machines available,
• for monitoring this PR brings a few prometheus metrics to check the status of the buffering pools
  For example:
  • we decide to have a pool of 4 buffers of type g4dn.xlarge,
  • autoscaling creates the said pool, these undergo different states (e.g. launched, ready-to-pre-pull-images, pulling images, stopping, ready to be used)
• for monitoring performance prometheus Histogram were also added in order to measure the times for:
  • launching a machine until it is ready to pre-pull
  • pre-pulling images

--> Next step will be adding these in the grafana dashboard

bonus:

• fixes Gauges not set back to 0 (they were cleared, which prometheus does not handle nicely)
• refactoring

Related issue/s

• fixes Autoscaling: add metrics for scaling timings (time to start machine, time to stop, time to create buffer, etc...) #6252, even if only partially is good for now
• fixes Autoscaling: add metrics for buffer pools #6251
• fixes Autoscaling: in non-billable deployments the maximum number of machines may be not respected for small amount of time #6123

How to test

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[codecov]

Codecov Report

Attention: Patch coverage is 98.47716% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.7%. Comparing base (cafbf96) to head (4292377).
Report is 483 commits behind head on master.

Files with missing lines	Patch %	Lines
...rvice_autoscaling/modules/instrumentation/_core.py	91.6%	2 Missing ⚠️
...e_service_autoscaling/modules/auto_scaling_core.py	75.0%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #6260      +/-   ##
=========================================
+ Coverage    84.5%   88.7%    +4.1%     
=========================================
  Files          10    1045    +1035     
  Lines         214   46592   +46378     
  Branches       25     406     +381     
=========================================
+ Hits          181   41352   +41171     
- Misses         23    5170    +5147     
- Partials       10      70      +60     

Flag	Coverage Δ
integrationtests	64.8% <ø> (?)
unittests	86.0% <98.4%> (+1.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ackages/aws-library/src/aws_library/ssm/_client.py	97.7% <100.0%> (ø)
...oscaling/src/simcore_service_autoscaling/models.py	100.0% <100.0%> (ø)
...e_autoscaling/modules/buffer_machines_pool_core.py	88.3% <100.0%> (ø)
...e_autoscaling/modules/buffer_machines_pool_task.py	100.0% <ø> (ø)
...ce_autoscaling/modules/instrumentation/__init__.py	100.0% <100.0%> (ø)
..._autoscaling/modules/instrumentation/_constants.py	100.0% <100.0%> (ø)
...autoscaling/modules/instrumentation/_ec2_client.py	100.0% <100.0%> (ø)
...ice_autoscaling/modules/instrumentation/_models.py	100.0% <100.0%> (ø)
...vice_autoscaling/modules/instrumentation/_utils.py	100.0% <100.0%> (ø)
.../src/simcore_service_autoscaling/utils/rabbitmq.py	100.0% <ø> (ø)
... and 2 more

... and 1029 files with indirect coverage changes

[odeimaiz]

👍

[mrnicegyu11]

left some minor comment but looks very nice and fancy thx

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud