Users: A way to limit number of jobs per user

[pacomf]

What is the expected behavior?

When we have keyclock integrated, we need to define a way to configure the maximum number of jobs allowed to run in our ray pods by user, to have a mechanism to control the saturation of the system.

We can have a number of jobs allowed to run at the same time by role, so depending of the user, we can provide more flexibility.

[IceKhan13]

One way to limit jobs per user is to create and assign cluster per user. Each cluster will have only 1 head node with min 1cpu, 1gb memory and scale up let's say to 2 workers with 4cpu and 4gb mem each. If user exceeds this capacity jobs will be sitting in a queue

[pacomf]

But in this case you need to know the users... or create and assign a new cluster every time that a new user is registered... so with an installation with thousands of users like IQP, it is a mess and over architecture :S

[pacomf]

But it can be a solution to discuss, for sure! we need to analyze it and take a decision :)

[IceKhan13]

Then maybe we need a small service to accept programs and then lunch them in cluster, which might be actually a better as we can control additional rules etc inside this service

[psschwei]

It might also make sense to consider limiting by total resource usage (memory / CPU requests), which you can do in kubernetes with resource quotas.

[IceKhan13]

Re-openning as mechanism is not there yet