Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛🎨Computational autoscaling: allow multi-machining/processing #5203

Merged
Changes from 1 commit
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
47194b9
instance ready is depending on mode
sanderegg Dec 21, 2023
ea6f013
hmm not yet totally there
sanderegg Dec 21, 2023
96114d5
missing manual testing parts
sanderegg Jan 9, 2024
e231f20
added remote debugging facilities
sanderegg Jan 9, 2024
8758ee9
added is_worker_connected and refcator
sanderegg Jan 9, 2024
0e48b85
added pending_nodes
sanderegg Jan 9, 2024
1b15221
use pending nodes and refactor
sanderegg Jan 9, 2024
7c62890
ensure negative values for cpus are capped to 0
sanderegg Jan 9, 2024
084e753
misconfigured in redis
sanderegg Jan 9, 2024
c53119b
renaming + logs
sanderegg Jan 9, 2024
b08beda
ensure not available scheduler does not break autoscaling
sanderegg Jan 9, 2024
fb21a6b
improving feedback
sanderegg Jan 9, 2024
885694a
do not expose redis
sanderegg Jan 9, 2024
a6de940
minor
sanderegg Jan 9, 2024
e876707
added function to get all processing tasks per worker
sanderegg Jan 9, 2024
47d7036
allow to max out the workers
sanderegg Jan 9, 2024
ad6c735
added worker retierement
sanderegg Jan 9, 2024
c7bcabb
use debugpy
sanderegg Jan 9, 2024
d1e4803
use debugpy
sanderegg Jan 9, 2024
7536eba
more robust
sanderegg Jan 9, 2024
fd8be0e
ensure stuff goes in the right location
sanderegg Jan 9, 2024
7e9d183
fix regex
sanderegg Jan 11, 2024
dbe2b2d
helper function
sanderegg Jan 11, 2024
c3e1d9b
improve logs
sanderegg Jan 11, 2024
3102058
fix dask fun issues
sanderegg Jan 11, 2024
c8acad8
refactor
sanderegg Jan 11, 2024
83f2e4e
typo
sanderegg Jan 11, 2024
5e64c0d
manual tester
sanderegg Jan 11, 2024
997aaab
mypy
sanderegg Jan 11, 2024
17f9720
docs
sanderegg Jan 11, 2024
6d59735
use Resources model
sanderegg Jan 12, 2024
92953cd
use resources, and make computing much faster by keeping available re…
sanderegg Jan 12, 2024
5df6fc2
type
sanderegg Jan 12, 2024
71867f4
simplify
sanderegg Jan 12, 2024
b82c7f1
refactor
sanderegg Jan 12, 2024
726d07c
typo
sanderegg Jan 12, 2024
06867f0
added delay for removing node
sanderegg Jan 12, 2024
15c904a
ruff
sanderegg Jan 13, 2024
e003438
missing parameter
sanderegg Jan 15, 2024
89415bd
keep task assignment in cluster
sanderegg Jan 15, 2024
9c33fa7
preparing to add assigned tasks in cluster
sanderegg Jan 15, 2024
c09b5e7
fix too many instances in
sanderegg Jan 15, 2024
4f2e729
allow to use replace
sanderegg Jan 15, 2024
939636c
refactor
sanderegg Jan 15, 2024
7341384
analyze with used resources
sanderegg Jan 15, 2024
5dc9183
in memory tasks only take memory
sanderegg Jan 15, 2024
edaf0cc
refactor
sanderegg Jan 15, 2024
ab66063
refactoring notifications
sanderegg Jan 15, 2024
69b6e8c
cleanup
sanderegg Jan 15, 2024
8aaa247
refactor
sanderegg Jan 15, 2024
e0794c5
fix clusters-keeper
sanderegg Jan 15, 2024
f839aed
refactoring
sanderegg Jan 15, 2024
71ba6b0
minor
sanderegg Jan 15, 2024
8ba7457
progress and logs now better
sanderegg Jan 15, 2024
ec899dc
improved notification mechanism
sanderegg Jan 16, 2024
c4b897b
reduce log pollution
sanderegg Jan 16, 2024
84aa9e5
syntax
sanderegg Jan 16, 2024
dd047a6
syntax
sanderegg Jan 16, 2024
80e9f09
fixed tests
sanderegg Jan 16, 2024
523d3c2
fixed comparison
sanderegg Jan 16, 2024
4dcb929
fixed confusing errors while running
sanderegg Jan 16, 2024
e46890b
only upgrade available resources if not set in constructor
sanderegg Jan 16, 2024
dc29dd7
tests are fixed here
sanderegg Jan 16, 2024
81e36cd
fixed tests
sanderegg Jan 16, 2024
61bafbe
fixed tests
sanderegg Jan 16, 2024
eb14d4b
mypy
sanderegg Jan 16, 2024
ebf85f6
removed unused code
sanderegg Jan 16, 2024
0e18cf1
remove catastrophic backtracking
sanderegg Jan 16, 2024
0441def
remove catastrophic backtracking
sanderegg Jan 16, 2024
cd99cd1
also check pending nodes
sanderegg Jan 16, 2024
a73f7b7
necessary for docker >24
sanderegg Jan 16, 2024
4b6f787
remove too many logs
sanderegg Jan 16, 2024
78e2869
only deactivate active nodes
sanderegg Jan 16, 2024
3c8d3ed
fixed regex
sanderegg Jan 16, 2024
e98b0e6
fixed test
sanderegg Jan 16, 2024
d6ef835
clean up
sanderegg Jan 16, 2024
d6d7b18
@pcrespov review: rename fct
sanderegg Jan 17, 2024
019d0bb
@pcrespov review: improve code
sanderegg Jan 17, 2024
c703f6f
@pcrespov review: refactor
sanderegg Jan 17, 2024
d23e109
@pcrespov review: remove unused fct
sanderegg Jan 17, 2024
b337597
@pcrespov review: add asserts to convey message
sanderegg Jan 17, 2024
40a4e25
@pcrespov review: refactor
sanderegg Jan 17, 2024
6787420
some typos
sanderegg Jan 17, 2024
a178756
mypy
sanderegg Jan 17, 2024
a1dc532
adapt to new naming
sanderegg Jan 17, 2024
5eb5c36
fix calls to dask_scheduler stuff
sanderegg Jan 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clean up
sanderegg committed Jan 16, 2024
commit d6ef83594081d0c66c7dcd30f5c813df3b131a16
14 changes: 0 additions & 14 deletions services/autoscaling/src/simcore_service_autoscaling/models.py
Original file line number Diff line number Diff line change
@@ -6,20 +6,6 @@
from models_library.generated_models.docker_rest_api import Node


@dataclass(kw_only=True, slots=True)
class AssignedTasksToInstance:
instance: EC2InstanceData
available_resources: Resources
assigned_tasks: list

def has_resources_for_task(self, task_resources: Resources) -> bool:
return bool(self.available_resources >= task_resources)

def assign_task(self, task, task_resources: Resources) -> None:
self.assigned_tasks.append(task)
self.available_resources -= task_resources


@dataclass(kw_only=True, slots=True)
class AssignedTasksToInstanceType:
instance_type: EC2InstanceType