Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛Bugfix/autoscaling does not scale above limit #5129

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Dec 4, 2023

What do these changes do?

During testing it was found that in the use case where a user starts 16 jobs:

  • no cluster workers are created because the autoscaling is currently set with EC2_INSTANCES_MAX_INSTANCES set to 10,
  • the autoscaling services completely refuses to create any instance

This PR changes that behavior by creating up to EC2_INSTANCES_MAX_INSTANCES instances (currently hard-coded to 10) so that the computation may still go through.

also includes:
#5137

Related issue/s

How to test

Dev Checklist

DevOps Checklist

@sanderegg sanderegg added this to the Kobayashi Maru milestone Dec 4, 2023
@sanderegg sanderegg self-assigned this Dec 4, 2023
@sanderegg sanderegg force-pushed the bugfix/autoscaling-does-not-scale-above-limit branch from 97fd641 to e6365cc Compare December 4, 2023 21:19
@sanderegg sanderegg marked this pull request as ready for review December 4, 2023 21:23
@sanderegg sanderegg added the a:autoscaling autoscaling service in simcore's stack label Dec 4, 2023
Copy link

codecov bot commented Dec 4, 2023

Codecov Report

Merging #5129 (0417a59) into master (7c30bc0) will increase coverage by 21.9%.
The diff coverage is 100.0%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #5129      +/-   ##
=========================================
+ Coverage    64.8%   86.8%   +21.9%     
=========================================
  Files         526     838     +312     
  Lines       26660   37406   +10746     
  Branches      198     209      +11     
=========================================
+ Hits        17292   32485   +15193     
+ Misses       9318    4871    -4447     
  Partials       50      50              
Flag Coverage Δ
integrationtests 64.9% <100.0%> (+<0.1%) ⬆️
unittests 84.7% <100.0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...c/dask_task_models_library/resource_constraints.py 100.0% <100.0%> (ø)
...simcore_service_director_v2/modules/dask_client.py 93.1% <100.0%> (+14.6%) ⬆️

... and 620 files with indirect coverage changes

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@sanderegg sanderegg force-pushed the bugfix/autoscaling-does-not-scale-above-limit branch from e1e133f to b12e742 Compare December 5, 2023 07:36
commit 96ba5f5
Author: sanderegg <[email protected]>
Date:   Tue Dec 5 10:19:31 2023 +0100

    use method to create hardware constraint and read
@sanderegg sanderegg force-pushed the bugfix/autoscaling-does-not-scale-above-limit branch from 0da3023 to 91e4ffa Compare December 5, 2023 09:43
Copy link

sonarqubecloud bot commented Dec 5, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

Copy link

codeclimate bot commented Dec 5, 2023

Code Climate has analyzed commit 0417a59 and detected 0 issues on this pull request.

View more on Code Climate.

@sanderegg sanderegg merged commit ec12cfc into ITISFoundation:master Dec 5, 2023
55 checks passed
@sanderegg sanderegg deleted the bugfix/autoscaling-does-not-scale-above-limit branch December 5, 2023 10:41
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Jan 8, 2024
33 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:autoscaling autoscaling service in simcore's stack
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants