Slow runtime on slurm cluster

[be-marc]

I am running 120 daemons on a slurm cluster. I have observed very unusual runtimes when .f uses functions from an external package. I use mlr3misc::map() in the example because there is an equivalent in the base package. I also tried completely different functions with the same result. The only thing they have in common is that they are not in the base package.

daemons(
  n = 120,
  url = host_url(ws = TRUE),
  remote = remote_config(
    command = "sbatch",
    args = c("--mem 512", "-n 1", "--wrap", "."),
    rscript = file.path(R.home("bin"), "Rscript"),
    quote = TRUE
  )
)

m = mirai_map(seq(120), function(i) {
  timestamp = Sys.time()
  mlr3misc::map(seq(10),function(j) j)
  difftime(Sys.time(), timestamp, units = "secs")
})

system.time({
  collect_mirai(m)
})

#  user  system elapsed 
# 0.006   0.006 165.554 

Most of the mirais are resolved after a few milliseconds. A few mirais needs seconds to finish and the last one 165 seconds.
The exact runtime varies of course but usually this small example takes more than a minute. A few times the measured runtimes in the mirais looked good but collect_mirai(m) still took a minute. What I have also observed is that if .f is a longer running function, about 90 mirias are started directly and then a new mirai starts every few seconds. I looked at a timestamp that is recorded directly at the beginning of .f.

If we use lapply from the base package instead of mlr3misc::map, the examples finishes in milliseconds.

m = mirai_map(seq(120), function(i) {
  timestamp = Sys.time()
  lapply(seq(10), function(j) j)
  difftime(Sys.time(), timestamp, units = "secs")
})

system.time({
  collect_mirai(m)
})

#  user  system elapsed 
# 0.001   0.001   0.000 

I suspect that the shared file system of the cluster is overloaded when 120 mirais try to load a package. When I add Sys.sleep(sample(10, 1)) to the beginning of .f , I always get a runtime of 10 seconds. So it seems to be enough to delay the loading of the packages. I tried to run the same .f with another parallelization framework called batchtools and I also got some expired jobs. So I don't think it's a problem caused by mirai but maybe mirai can solve it?

[be-marc]

I tried a bit more. Calling everywhere({library(mlr3misc)}) before, results in all following mirais to evaluate in a few milliseconds. But it takes a minute until all daemons evaluate everywhere({library(mlr3misc)}). So the problem is only shifted. What I have also observed is that starting the daemons is also very slow from about 100 daemons. Slurm already reports the jobs as running but the daemons did no call back to the main session. It is probably the same problem but this time with loading the R session and base packages.

[shikokuchuo]

It would be great if you can test with the latest dev version of mirai 1.3.1.9022 using a non-websocket host URL (this is now supported - changelog). Just to rule out the possibility that it's a networking issue. Thanks.

[be-marc]

Unfortunately, it doesn't make a difference.

[be-marc]

I tested it again on another cluster. The cluster is much larger. You can easily use 100,000 cpu hours per day there. I was able to run mlr3misc::map in 1000 jobs at the same time. Loading the package took a maximum of 3 seconds. The problem therefore seems to depend very much on the performance of the file system.

I talked to the developer of batchtools and he told me that this is a known issue. He added a random sleep to each job.

[shikokuchuo]

I don't think this falls under the remit of mirai as it's a general tool used in lots of different situations.

It would logically be better implemented where this is known to be an issue e.g. cluster managers if this is common on large cluster systems, or if it does depend entirely on the filesystem then perhaps it does fall to user code to include a random sleep at the top of the code.