Dask backend: Create a separate cluster for group automatically

[sanderegg]

Currently one has to manually create a separate machine in AWS where:

• a docker swarm is created
• a osparc-dask-gateway is started

Once this is done, the use can register the osparc-dask-gateway in oSparc GUI and send computational jobs to that cluster.
Later after use, the cluster is manually terminated.

This needs to be automated. ideally the cluster shall remain up for some time after last use.

Tasks

Preview Give feedback

1. ✨Meet the clusters-keeper service (⚠️ devops) #4591
   a:clusters-keeper +1
   
2. Comp backend/clusters creation functionalities #4602
   a:clusters-keeper +1
   
3. Rabbitmq/separate clients #4685
   a:services-library +1
   
4. ✨RabbitMQ: Add RPC decorator for server side #4682
   a:services-library +1
   
5. ✨Computational backend: connect dv2 to clusters keeper for on-demand clusters (🗃️ + ⚠️devops) #4703
   a:clusters-keeper a:director-v2 +2
   
6. ♻️Rabbitmq rpc: response shall not be jsonized #4730
   a:clusters-keeper a:services-library +2
   
7. ✨New groups extra properties to use on demand clusters in DB (🗃️) #4736
   a:database +1
   
8. ✨Enable on-demand clusters based on groups extra properties (⚠️ devops) #4738
   a:webserver +1
   
9. ♻️On-demand clusters: remove dask-gateway #4760
   a:clusters-keeper +1
   
10. ✨Api-server: allow usage of on-demand clusters if defined in groups_extra_properties #4787
    a:apiserver +1
    

[sanderegg]

Proposal

• Create a clusters-manager service in osparc-simcore

  • responsible to create and terminate external clusters in AWS
  • a cluster may be auto-managed (with an autoscaling) or fixed size?
  • a cluster is made of at least 1 computer
  • a cluster is a docker swarm
  • a cluster has a dask-gateway running on the computer
  • a cluster shall have access to the osparc registry and dockerhub (need the correct network setup)
  • a cluster shall be shutdown when it is not used anymore (heartbeat? find out if still in use?)

• oSparc GUI shall allow the creation of the cluster (in the current clusters preferences?)

  • user shall decide which type of instances (small, medium, big), number of instances? or max number of instances?
  • user can share the cluster (already available I think)
  • the ID of the cluster shall be visible somewhere, for use with the PublicAPI

• similar options shall be available in PublicAPI

[sanderegg]

Implementation completed during The Nameless sprint.

a cluster is created for user/wallet combination for now.
a cluster is:

• 1 machine is created through EC2
• the machine is the manager of a docker swarm
• the machine currently runs a dask-scheduler/dask-worker
• the machine is created through RPC call to the clusters-keeper service
• the clusters-keeper service terminates the machine once it is deemed as unused for more than 5 minutes