Timeout when initializing SSH tunnel

[brendanf]

I'm trying to submit jobs to my cluster (SLURM) via SSH in drake with the clustermq backend. I need to load a conda environment on the cluster before running R. I modified the template to activate the conda environment (and also set the R profile):

"conda activate <my_environment> && \
        cd {{ project_dir | <my_directory> }} && \
        R_PROFILE={{ remote_profile | config/slurm.Rprofile }} R --no-save --no-restore -e \
        'clustermq:::ssh_proxy(ctl={{ ctl_port }}, job={{ job_port }})' \
        > {{ ssh_log | /dev/null }} 2>&1"

This failed with error message Error in .subset2(public_bind_env, "initialize")(...) : Remote R process did not respond after 5 seconds. Check your SSH server log.

Checking the SSH server log as suggested:

R version 3.5.1 (2018-07-02) -- "Feather Spray"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> clustermq:::ssh_proxy(ctl=54344, job=50552)
master ctl listening at: tcp://localhost:54344
forwarding local network from: tcp://rackham2:8266
sent PROXY_UP to master ctl

I assume this is what the output should look like. I'm guessing that the issue is that loading the conda environment and starting R is taking longer than 5 seconds, so the initialization is timing out locally. Apparently this 5 second timeout is hardcoded. Would it be possible to expose it as an option?

[brendanf]

I've tried this a few more times; it sometimes works, suggesting that it really is a timing issue.

[mschubert]

Hi @brendanf 👋 ,

Thank you for spotting this!

Your reasoning makes sense, there may just not be enough time to activate and environment and start R within the timeout given.

Do you want to try and send a patch for this?

[brendanf]

OK, coming up. Is there any documentation of package options that I should add this to? I haven't been able to find a single place for them.

[mschubert]

A lot of documentation on the options is still lacking, but I'm holding off on that until I structure them better (0.9, 1.0)

[mschubert]

Thank you for #158, I just merged it.

Does this solve your problem here as well?

[brendanf]

Does this solve your problem here as well?

Yes, although I added some documentation for the change in #159, and ropensci/drake#933 is still a (separate) issue with the SSH backend.