using --cores

[igordot]

I am trying to understand the multi-core usage instructions. It says for --cores 2 would actually use up to 9 cores (2 input, 2 output, 4 cutadapt, 1 trim galore). What happens when you set --cores 1? By the previous logic, it should actually be 5 cores. Or does that only apply if you specify multiple cores?

If you have less than 9 cores available, does that mean you should not be using multi-threaded option?

[FelixKrueger]

Hi @igordot

This was discussed quite extensively over here: nf-core/atacseq#65 (comment)

The main part from that thread is probably:

...in default mode (--cores 1) I would expect 2 cores at high load, and up to 2 cores for (de-)compression, albeit at negligible load for both SE and PE. --cores 2 would then formally require 5 cores, --cores 3 6, --cores 4 7 etc.

If you are the only person using that machine then it should be fine to use a little more. Just generally, if you over-commit your CPU with processes then you will start to slow down all processed as they are competing for resources. They will probably still finish fine, but it's not really ideal. Does that make sense?

[igordot]

Thank you for pointing out that discussion. I don't think I would have found it myself.

Can you clarify what you mean by "formally"? I am not sure why it goes from 9 to 5 for --cores 2.

[FelixKrueger]

Well, if you expressed the CPU usage from low to high as 0 to 100% per core, then 2 cores would run at 100%, and 2 cores at the maybe 5% each, giving a total of 210% for a standard run. If you submitted this to a queue system that very strictly monitors the number of cores used, the number would be 4, even thought 'real' usage would only be 2 and a bit.

If your 9 core system belongs to you only, you can probably use --cores 4 without any issues.

[igordot]

Thank you for clarifying.