new nested params load feature breaks workflow

[odoublewen]

@lukfor The support for nested params and loading them from external files (#50, fixing #15) is a very welcome new feature, thank you!

However, I am confused by something -- it seems like the load() method is injecting baseDir and outputDir into my params. And not just at the top level but in nested maps too. Example:

Here is my main.nf.test file:

nextflow_pipeline {
    name "Pipeline Quick Integration Test main.nf"
    script "main.nf"
    test("Should run without failures") {
        when {
            params {
                load("$baseDir/tests/params/params_debug.json")
            }
        }
        then {
            println(params)
            assert workflow.success
        }
    }
}

Here is my params_debug.json file:

{
  "references": {
    "GRCh38": {
      "description": "Homo sapiens (GRCh38.p13)",
      "bwa_index": "path/to/GRCh38.fa.bwt",
    }
  },
  "output_dir": "tests/output/smoke/",
  "samples": [
    {
      "id": "SAMP1",
      "name": "Sample 1",
      "filepath_r1": "tests/data/reads/SAMP1_R1_001_10k.fastq",
      "filepath_r2": "tests/data/reads/SAMP1_R3_001_10k.fastq",
      "filepath_umi": "tests/data/reads/SAMP1_R2_001_10k.fastq",
      "reference": "GRCh38"
    }
  ]
}

Here is the (pretty printed) output of that println call in my main.nf.test file:

[
  baseDir:/work/mypipeline,
  outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
  references: [
    baseDir:/work/mypipeline,
    outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
    GRCh38: [
      baseDir:/work/mypipeline,
      outputDir:/work/mypipeline/.nf-test/tests/42062baab057bce9c1028ff0ac9ef88b/output,
      description:Homo sapiens (GRCh38.p13),
      bwa_index:path/to/GRCh38.fa.bwt
    ]
  ],
  output_dir:tests/output/smoke/,
  samples:[
    [
      id:SAMP1,
      name:Sample 1,
      filepath_r1:tests/data/reads/SAMP1_R1_001_10k.fastq,
      filepath_r2:tests/data/reads/SAMP1_R3_001_10k.fastq,
      filepath_umi:tests/data/reads/SAMP1_R2_001_10k.fastq, reference:GRCh38
    ]
  ]
]

The thing is, my workflow expects params.references to be a map with a particular structure. So the injection of baseDir and outputDir makes my workflow choke.

It seems like this injection into nested maps is by design, but I am left scratching my head. Is it really necessary to do this to all nested maps? I imagine this could break any workflow that makes assumptions about nested maps structure.

(cc: @birnbera)

[lukfor]

Thanks for reporting this issue 👍 I will look into it and remove the variables from the nested maps.

[aaron-fishman-achillestx]

@lukfor perhaps you can do this by putting these vars in the delegate of the closures, rather than in the closure's themselves?

[aaron-fishman-achillestx]

or could be manually removed as well I guess.

[lukfor]

@lukfor perhaps you can do this by putting these vars in the delegate of the closures, rather than in the closure's themselves?

I remember I had some problems with this approach and variable substitution. But I will look into that the next days.

[birnbera]

As a workaround, I found that I can replicate the load functionality without the extra keys side effect using:

params {
    delegate.putAll(path('path/to/params.json').json)
    additional_key = 'some value'
}

I believe this works because the params map is the delegate of the params closure and forwards the putAll method to the underlying map.

[lukfor]

PR #117 will fix also this issue.