Question: global way to always print to stdout/stderr?

[vsoch]

Apologies if this is a noob question! I am testing the machine learning workflow: https://github.com/nextflow-io/ml-hyperopt/
and have gotten it to be able to run with the Flux Executor - as expected, nextflow kicks off a bunch of jobs!

Of course then I quickly realized they don't have terminal output (e.g., the flux process can show this when you click on the job) and first I thought it was because I specified --output for the FluxExecutor, but now I'm realizing that nextflow can handle different kinds of output, and it's usually defined in the progress (e.g., I found an example here https://gitter.im/nextflow-io/nextflow?at=5afa7ecfd245fe2eb7af6dc5 and the docs here https://www.nextflow.io/docs/edge/process.html#outputs). It looks like the specification for where to write output, in the case of this workflow, happens in the modules: https://github.com/nextflow-io/ml-hyperopt/tree/master/modules. I also found "debug" and I think that is of the same thread - it needs to be defined in each workflow file. Given something that already exists, I've have to custom write each of the files to better control the output.

So my question - which I hope there is a stupidly simple answer to - is if there is a way to globally say "Always print your texty stuffs to stdout/stderr" and that way the Flux RESTFul API interface can show it easily. If there is a way to control it within the FluxExecutor (and I'm able to add some special flag) that would be great to know too - basically I just want stuff to print to the terminal within the process that is running the job. Thank you!

[mribeirodantas]

Instead of adding debug true to the beginning of every process block, you can set it once in the nextflow.config file. Simply adding the line below to the nextflow.config file in your launching directory...

process.debug = true

will make the following Nextflow script print hello and then world to the standard output.

process A {

  output:
    stdout
  script:
  """
  echo 'hello'
  """
}

process B {

  output:
    stdout
  script:
  """
  echo 'world'
  """
}

workflow {
  A()
  B()
}

Try removing the line from the nextflow.config file and you'll see the hello and world will no longer be printed to the standard output.

There is also another way to do that, without having to create/change the nextflow.config file. Simply add -process.echo to the command line. For example:

➜  ~ nextflow run ex2.nf -process.echo

One last trick! Even if Nextflow doesn't print to your standard output, it must print somewhere, right? You can see the output by checking the content of the .command.out file that is located within every task folder (try typing tree work where you launched your pipeline to see the folders). If you have just run the pipeline I wrote above without seeing the hello and world, you can see them by typing the following command:

➜  ~ nextflow log last | while read line ; do cat $line/.command.out ; done

[vsoch]

At the end of the day, we'd need a way for the user to request this functionality from the Nextflow config, either in the config or one off. I'm not sure how a function like that would help if we aren't allowed to do that. This is also a web server - we aren't really able to run commands like tee willy nilly to get logs - the way we get logs consistently across any command launched is with the flux jobs attach. I can't be implementing custom logic just for Nextflow.

[pditommaso]

I've replied in the PR #3424 (comment)

[bentsherman]

Thanks @grondo for chiming in.

Otherwise, perhaps "wrapping" the nextflow jobs submitted to Flux with a tee seemed like a workable solution for this case. Actually, I wonder if we could add a mode to the job shell that would redirect to a file and the kvs output eventlog. Feel free to open an issue if this would be a nice feature.

It took me a while to grok the whole problem here, but this is the solution I settled on at the end of the day. Flux should just always send the process output to stdout, but then also to a file if --output is specified. I know PBS works this way, possibly SLURM as well. I suspect you will eventually have more users who want the terminal output and the log file, not just for Nextflow but any other tool that works this way.

Anyway, for now I think we will just merge @vsoch 's PR since it is an opt-in feature.

[bentsherman]

@vsoch I apologize for starting out by saying your PR probably wouldn't be merged, that was rash of me 😄

[grondo]

I know PBS works this way, possibly SLURM as well. I suspect you will eventually have more users who want the terminal output and the log file, not just for Nextflow but any other tool that works this way.

Great advice! Though I'm pretty sure Slurm can't do this (former Slurm developer here). We should be able to support this mode though, and I can see it will definitely be useful. 👍