Issue running with Singularity on SLURM

[drejom]

Prework

• Read and agree to the Contributor Code of Conduct and contributing guidelines.
• Confirm that your issue is a genuine bug in the crew.cluster package itself and not a user error, known limitation, or issue from another package that crew.cluster depends on.
• If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• Post a minimal reproducible example like this one so the maintainer can troubleshoot the problems you identify. A reproducible example is:
  • Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • Readable: format your code according to the tidyverse style guide.

Description

After updating a slew of packages recently, my SLURM-enabled targets pipeline has stopped running, with errors about seconds_timeout. I have a rather elaborate script to setup cluster operations, but I think I've narrowed it down to crew_controller_slurm(), so only post that here:

Reproducible example

# working on it

Apologies @wlandau my initial example was not in fact reproducible, but while I see if I can make a minimal example, does this {targets} error give any clues as to what's going on? It occurs with or without seconds_timeout set in crew.cluster::crew_controller_slurm()

Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    all(is.numeric(.)) && all(length(.) == 1L) && all(!anyNA(.)) && all(. >= 0) is not true on . = seconds_timeout

Diagnostic information

sessionInfo()
R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Los_Angeles
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] crew.cluster_0.2.0 targets_1.4.1      lubridate_1.9.3    forcats_1.0.0      stringr_1.5.1      dplyr_1.1.4        purrr_1.0.2        readr_2.1.4        tidyr_1.3.0       
[10] tibble_3.2.1       ggplot2_3.4.4      tidyverse_2.0.0   

loaded via a namespace (and not attached):
 [1] utf8_1.2.4         generics_0.1.3     stringi_1.8.3      hms_1.1.3          digest_0.6.33      magrittr_2.0.3     grid_4.3.1         timechange_0.2.0   jsonlite_1.8.8    
[10] processx_3.8.3     backports_1.4.1    ps_1.7.5           fansi_1.0.6        scales_1.3.0       crew_0.8.0         codetools_0.2-19   cli_3.6.2          rlang_1.1.2       
[19] munsell_0.5.0      withr_2.5.2        yaml_2.3.8         parallel_4.3.1     tools_4.3.1        tzdb_0.4.0         getip_0.1-4        nanonext_0.11.0    colorspace_2.1-0  
[28] base64url_1.4      vctrs_0.6.5        R6_2.5.1           lifecycle_1.0.4    pkgconfig_2.0.3    callr_3.7.3        pillar_1.9.0       gtable_0.3.4       glue_1.6.2        
[37] data.table_1.14.10 xfun_0.41          tidyselect_1.2.0   knitr_1.45         mirai_0.11.3       igraph_1.6.0       compiler_4.3.1    

[drejom]

Ok, so downgrading {targets} to 1.2.2 solved things for now and I can run my analysis.

However, I see a number of changes in 1.3.0 which i suspect account for the error.

I can run the targets-minimal pipeline without issue, but when i include the following to run it on SLURM, I get errors.

nodename <- Sys.info()["nodename"]

singularity_exec <- glue::glue("cd {here::here()} \\
/{base_dir}/easy-build/software/singularity/3.7.0/bin/singularity exec \\
--env R_LIBS_USER=~/R/bioc-3.17 \\
--env R_LIBS_SITE=/{base_dir}/singularity/shared_cache/rbioc/rlibs/bioc-3.17 \\
-B /{base_dir}/singularity,/ref_genomes,/scratch \\
/{base_dir}/singularity/shared_cache/rbioc/vscode-rbioc_3.17.sif \\")

slurm <- crew.cluster::crew_controller_slurm(
    host = nodename,
    script_lines = singularity_exec)

tar_option_set(
    controller = slurm,
    resources = tar_resources(
        crew = tar_resources_crew(seconds_timeout = 3)
        )
    )

targets::tar_make()
▶ dispatched target raw_data_file
▶ completed pipeline [6.776 seconds]
Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    target NA error: 'errorValue' int 5 | Timed out

If I remove the resources section from tar_option_set():

resources = tar_resources(
        crew = tar_resources_crew(seconds_timeout = 3)
        )

I get no error, but the pipeline never progresses beyond dispatching the first target:

targets::tar_make()
▶ dispatched target raw_data_file
/

Apologies if I'm missing something obvious, but are you able to provide any insight?

[wlandau]

First I would try upgrading mirai and nanonext to the latest versions on CRAN (both 0.12.0). That may solve the errors/hanging with away and allow you to upgrade targets as well.

It looks like the mirai dispatcher process may be taking a long time to either start (you could confirm by timing how long it takes to start a new R process on your setup), or you may have high-memory objects which are taking a long time to send to the dispatcher and workers. For this, you could raise the seconds_timeout in the controller, and I would not recommend setting seconds_timeout in tar_resources_crew(). The latter is different: it is for timeouts in tasks, and 3 seconds is very short for this.

But this is just speculation. If upgrading packages does not work, we will need to work with a fully reproducible/shareable test case in order to move forward.

[drejom]

Thanks @wlandau, upgrading all packages and slimming down the config seems to have got things back on track. I've now managed to reproduce the error I had earlier. Here's a minimal example:

library(targets)
library(crew)
library(crew.cluster)

## local ---------------------------------------------------
tar_script({
    tar_option_set(controller = crew::crew_controller_local())
    list(
        tar_target(y1, 1 + 1),
        tar_target(y2, 1 + 1),
        tar_target(z, y1 + y2)
    )}, ask = FALSE)
tar_make(c("y1", "y2"))

## slurm ---------------------------------------------------
tar_script({
    singularity_bin <- [/path/to/singularity]
    base_dir <- [path/to/base/dir] 
    nodename <- Sys.info()["nodename"]
    singularity_exec <- glue::glue("cd {here::here()} \
{singularity_bin} exec \\
-B {base_dir},/scratch \\
--env R_LIBS_SITE={base_dir}/rlibs/bioc-3.17 \\
{base_dir}/vscode-rbioc_3.17.sif \\")
    slurm <- crew.cluster::crew_controller_slurm(
        host = nodename,
        script_lines = singularity_exec)
    tar_option_set(controller = slurm)

    list(
        tar_target(y1, 1 + 1),
        tar_target(y2, 1 + 1),
        tar_target(z, y1 + y2)
    )}, ask = FALSE)
tar_make()

This works: the first two targets are submitted locally by crew, the 3rd is dispatched to the cluster and returns its result as expected.

My issue seems to be with controller groups:

## slurm - multiple controllers --------------------------------
tar_script({
    singularity_bin <- [/path/to/singularity]
    base_dir <- [path/to/base/dir] 
    nodename <- Sys.info()["nodename"]
    singularity_exec <- glue::glue("cd {here::here()} \
{singularity_bin} exec \\
-B {base_dir},/scratch \\
--env R_LIBS_SITE={base_dir}/rlibs/bioc-3.17 \\
{base_dir}/vscode-rbioc_3.17.sif \\")

    slurm_small <- crew.cluster::crew_controller_slurm(
        name = "small",
        host = nodename,
        slurm_cpus_per_task = 2,
        script_lines = singularity_exec)
    slurm_medium <- crew.cluster::crew_controller_slurm(
        name = "medium",
        host = nodename,
        slurm_cpus_per_task = 6,
        script_lines = singularity_exec)

    tar_option_set(
        controller = crew::crew_controller_group(slurm_small, slurm_medium),
        resources = tar_resources(
            crew = tar_resources_crew(controller = "small")))

    list(
        tar_target(y1, 1 + 1),
        tar_target(y2, 1 + 1),
        tar_target(z, y1 + y2)
    )}, ask = FALSE)
    tar_make()

This throws the error about seconds_timeout, as I found above:

▶ dispatched target y1
▶ dispatched (pending) target y2
▶ completed pipeline [1.802 seconds]
Error:
! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    all(is.numeric(.)) && all(length(.) == 1L) && all(!anyNA(.)) && all(. >= 0) is not true on . = seconds_timeout

Abridged sessionInfo():

 [1] crew.cluster_0.2.0 crew_0.8.0         targets_1.4.1      nanonext_0.12.0    mirai_0.12.0 

[drejom]

Awesome, thanks so much!!

[kyle-messier]

@drejom Did the ## slurm - multiple controllers setup work for you?

[drejom]

Yep, all works using the config as above

[mitchellmanware]

@drejom Did you have to use any additional commands after the vscode-rbioc_3.17.sif file in the singularity_exec script lines to properly dispatch the target? When I end script_lines parameter with the container image I get an error expecting additional arguments to follow.

_targets.R

#############################      CONTROLLER      #############################
default_controller <- crew.cluster::crew_controller_slurm(
  name = "default_controller",
  workers = 4,
  seconds_idle = 30,
)
gpu_scriptlines <- glue::glue(
  "#SBATCH --output=slurm/devgpu_%j.out \
  #SBATCH --error=slurm/devgpu_%j.err \
  apptainer exec \
    container_models.sif \\"
)
gpu_controller <- crew.cluster::crew_controller_slurm(
  name = "gpu_controller",
  host = gpu_host,
  workers = 4,
  seconds_idle = 30,
  options_cluster = crew.cluster::crew_options_slurm(
    verbose = TRUE,
    script_lines = gpu_scriptlines
  )
)
targets::tar_option_set(
  controller = crew::crew_controller_group(
    gpu_controller,
    default_controller
  ),
  resources = targets::tar_resources(
    crew = targets::tar_resources_crew(controller = "gpu_controller")
  )
)

list(
  targets::tar_target(
    t1,
    command = 1 * 1
  )
  ,
  targets::tar_target(
    t2,
    command = t1 + 1
  )
)

Output of initial tar_make() shows that separate batch script is submitted for target t1:

▶ dispatched target t1
Submitted batch job 481149

Additionally, the output is properly in the "slurm/devgpu_%j.out" folder, so some of the custom settings are working as expected. Contents of "slurm/devgpu_481149.out":

Error for command "exec": requires at least 2 arg(s), only received 0

Usage:
  apptainer [global options...] exec [exec options...] <container> <command>

Run 'apptainer --help' for more detailed usage information.
/var/spool/slurm/slurmd/job481149/slurm_script: line 9: container_models.sif: command not found

@kyle-messier @sigmafelix

[mitchellmanware]

I was misusing the glue function. The following script lines submit gpu-enabled targets to SLURM from within the Apptainer container.

#############################      CONTROLLER      #############################
default_controller <- crew::crew_controller_local(
  name = "default_controller",
  workers = 4,
  seconds_idle = 30,
)
gpu_scriptlines <- glue::glue(
  "#SBATCH --job-name=devgpu \
  #SBATCH --partition=partition_name \
  #SBATCH --gres=gpu:1 \
  #SBATCH --output=slurm/devgpu_%j.out \
  #SBATCH --error=slurm/devgpu_%j.err \
  apptainer exec --nv ",
  "--bind $PWD:/mnt --bind $PWD/_targets:/opt/_targets ",
  "container_models.sif \\"
)
gpu_controller <- crew.cluster::crew_controller_slurm(
  name = "gpu_controller",
  workers = 1,
  seconds_idle = 30,
  options_cluster = crew.cluster::crew_options_slurm(
    verbose = TRUE,
    script_lines = gpu_scriptlines
  )
)
targets::tar_option_set(
  controller = crew::crew_controller_group(
    default_controller,
    gpu_controller
  ),
  resources = targets::tar_resources(
    crew = targets::tar_resources_crew(controller = "default_controller")
  )
)

list(
  targets::tar_target(
    t1,
    command = 1 * 1
  )
  ,
  targets::tar_target(
    t2,
    command = t1 + 1
  )
  ,
  targets::tar_target(
    t3,
    command = torch::cuda_device_count(),
    resources = targets::tar_resources(
      crew = targets::tar_resources_crew(controller = "gpu_controller")
    )
  )
  ,
  targets::tar_target(
    t4,
    command = list(
      paste0("t1: ", t1),
      paste0("t2: ", t2),
      paste0("torch::cuda_device_count(): ", t3))
  )
)

@drejom Your example was very helpful in solving this issue, thanks for posting!