Optimize computational scheduler for 100s of jobs

[sanderegg]

The computational scheduler in the director-v2 shall be able to handle at least 1000 jobs concurrently without complaining.
This can be tested by using the osparc API and following the pseudo-code to maximize the load:

• have a cluster able to run 100 job at the same time (that imply 1 core per job since the dask-workers are finding, could be 10 computers with 10 cores)

• create 200 jobs

• start all jobs concurrently

• await their result

• monitor the load on the following services:

  • director-v2
  • dask-scheduler
  • dask-worker(s)
  • dask-gateway
  • rabbit
  • wb-db-event-listener

The services should still be responsive (e.g. it should be possible to run some pipelines through the GUI for example)

Tasks

Preview Give feedback

No tasks being tracked yet.

[sanderegg]

stale, and is currently being directly tested by users.