Limiting the number of cores?

[LunavdL]

Hi,

I am running porefile on a server with 24 cores, but when I start the pipeline the load of the system quickly increases to >30 or 40.
Is there a way to limit the maximum number of cores used by the pipeline?

Best regards,
Luna

[iferres]

Hi! Not sure if I understand your issue. How come it is using more than 30 cpus if your server has only 24?

The idea of nextflow is to run pipelines with a workload manager in the context of an hpc cluster or in the cloud, although it runs ok in a bare pc or server. The paradigm, however, is different as you set resources per process, and instances of processes are called as needed (as its inputs are available) and as the compute resources are available. You don't set a maximum limit of cpus as in traditional software or scripts.

You need to create the configuration for your infrastructure. I can't help very much here. In the processes.nf script you can see the processes defined in porefile. You will see that some of them have a label 'big_cpu' of label 'big_mem' tag. You can define a common configuration for each label, or define a configuration independently for each process. See nagual.config for a minimal example of configuration you can use as guide. Have a look at nextflow's configuration documentation. Also process directives may help you, probably using maxForks directive is a good idea in your case, or limiting the global number of instances using -queue-size from the command line.

I hope you find these links useful, and sorry I can't provide a simpler answer.

[LunavdL]

Hi,

Thank you again for your reply!
The server has only 24 CPUs, but the CPU load can increase to much more than that (although a healthy CPU load should not exceed the number of CPUs). We looked at a few log reports and the CPU load increased to almost 200.

I will have a look at the configuration documents and see if I can limit the load in that way.

Best regards,
Luna

[iferres]

Actually, if you are using the --minimap2 workflow you could benefit from the nagual configuration, as it launches only 1 Minimap2 process at a time (uses the maxForks = 1 directive), which may be the most intensive resource one. This profile uses singularity as container engine. We use it in a small server we have, similar as the one you are describing I think.

nextflow run microgenlab/porefile --minimap -profile nagual ...