Pipeline with multiple input and multiple output

[zillurbmb51]

Hi, I am a new user of nextflow. I have test1.bed, test1.bim, test1.fam files. I want to do some qc using plink in nextflow. How can I import those three files in nextflow and get output test2.bed, test2.bim, test2.fam which will be used in the next step to generate test3.bed, test3.bim, test3.fam and so on? I also need to save all the intermediate files in a directory. Any help with example?
Kind regards, Zillur

[abhi18av]

Hi @zillurbmb51 ,

Would it be possible for you to re-phrase the question or provide a snippet of code (something we can debug) and fix?

[zillurbmb51]

Thank you @abhi18av
My original operation in bash looks like the following:
`#Step 1:
plink --bfile test1 --maf 0.05 --make-bed --out test2

#Step 2:
plink --bfile test2 --geno 0.02 --make-bed --out test3

#Step 3:
plink --bfile test3 --maf 0.01 --make-bed --out test4
#Continue with more steps`
Here test1 means test1.bed, test1.bim and test1.fam. test2 will be test2.bed, test2.bim and test2.fam. test3 will be test3.bed, test3.bim and test3.fam .
I want to convert this pipeline from bash to nextflow. Any help, if possible example. Best, Zillur

[abhi18av]

Hi @zillurbmb51 ,

Perhaps this might be of use in this case (it's a matter of contextual design)

process PLINK {
input:
path(test1)

output:
path(test4)

script:
"""

#Step 1:
plink --bfile ${test1} --maf 0.05 --make-bed --out test2

#Step 2:
plink --bfile ${test2} --geno 0.02 --make-bed --out test3

#Step 3:
plink --bfile ${test3} --maf 0.01 --make-bed --out test4
"""

}

Next thing to consider,

Here test1 means test1.bed, test1.bim and test1.fam. test2 will be test2.bed, test2.bim and test2.fam. test3 will be test3.bed, test3.bim and test3.fam ...

Depending on the exact extension/files you need as input or output, you can update the input and output directive in the above Nextflow process (here's the doc).

[zillurbmb51]

Thank you @abhi18av
I was trying to use filePairs option for input but getting the attached error. I was trying different ways, but none is working. Any help?

[zillurbmb51]

Here is the script and error report:
`params.dir="data1/"
dir=params.dir
params.publish="output/"

Channel
.fromFilePairs
("{params.dir}*.{bed,fam,bim}",size:3)
.ifEmpty {error "No matching plink files"}
.set {plink_data}

process maf {
input:
set file (pl_files) from plink_data
output:
files test1 into result1
publishDir params.publish
script:
plink --bfile $pl_files --maf 0.05 --make-bed --out $test1
}
process gen {
input:
set file (test1) from result1
output:
pulishDir params.publish
files test2 into result2
script:
plink --bfile $test1 --geno 0.05 --make-bed --out $test2
}
process maf2 {
input:
set file (test2) from result2
output:
publishDir params.publish
set file (test3) into result3
script:
plink --bfile $test2 --maf 0.01 --make-bed --out $test3
}
Error report:ubuntu@ubuntu-Precision-7730:~/nextflowtest$ nextflow run test1.nf.txt
N E X T F L O W ~ version 20.10.0
Launching test1.nf.txt [big_coulomb] - revision: 4107489075
Script compilation error

• file : /home/ubuntu/nextflowtest/test1.nf.txt
• cause: Unexpected input: ',' @ line 7, column 31.
  ("{params.dir}*.{bed,fam,bim}",size:3)
  ^

1 error
`

[abhi18av]

Hi @zillurbmb51 ,

Without running the script (as I don't have the files), I can only highlight the issues regarding

• usage of $ in fromFilePairs
• use of correct ordering of publishDir (before the input/output)
• usage of """ in the script directive

params.dir="data/"
params.publish="output/"

Channel
.fromFilePairs("${params.dir}.{bed,fam,bim}", size:3)
.ifEmpty {error "No matching plink files"}
.set {plink_data}

process maf {
publishDir params.publish

input:
set file (pl_files) from plink_data

output:
files test1 into result1

script:
"""
plink --bfile $pl_files --maf 0.05 --make-bed --out $test1
"""
}

process gen {
publishDir params.publish

input:
set file (test1) from result1

output:
files test2 into result2

script:
"""
plink --bfile $test1 --geno 0.05 --make-bed --out $test2
"""
}

process maf2 {
publishDir params.publish


input:
set file (test2) from result2

output:
set file (test3) into result3

script:
"""
plink --bfile $test2 --maf 0.01 --make-bed --out $test3
"""
}

[abhi18av]

I suggest that it's worth looking into the Nextflow basic concepts for avoiding syntactic issues.

In addition, for effective communication regarding debugging NF code, perhaps its also worth giving a look to Markdown.

[zillurbmb51]

Hi @abhi18av
Thank you very much. I will follow the given link. However, I am getting a new error:

ubuntu@ubuntu-Precision-7730:~/nextflowtest$ nextflow run test1.nf N E X T F L O W ~ version 20.10.0 Launching test1.nf[golden_leakey] - revision: 2b119e756f WARN: Inputsetmust define at least two component -- Check processmafWARN: Outputsetmust define at least two component -- Check processmaf[- ] process > maf - [- ] process > gen - [- ] process > maf2 - WARN: Inputsetmust define at least two component -- Check processgen[- ] process > maf - [- ] process > gen - [- ] process > maf2 - WARN: Inputsetmust define at least two component -- Check processgenWARN: Outputsetmust define at least two component -- Check processgenWARN: Inputsetmust define at least two component -- Check processmaf2WARN: Outputsetmust define at least two component -- Check processmaf2WARN: Input tuple does not match input set cardinality declared by processmaf` -- offending value: [test1, [/home/ubuntu/nextflowtest/data1/test1.bed, /home/ubuntu/nextflowtest/data1/test1.bim, /home/ubuntu/nextflowtest/data1/test1.fam]]
Error executing process > 'maf (1)'

Caused by:
Unknown variable 'test1' -- Make sure it is not misspelt and defined somewhere in the script before using it

Source block:
"""
plink --bfile $pl_files --maf 0.05 --make-bed --out $test1
"""

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out
`

But I did define test1 before using, right? here:

`params.dir="data1/"
params.publish="output/"

Channel
.fromFilePairs("${params.dir}*.{bed,fam,bim}",size:3)
.ifEmpty {error "No matching plink files"}
.set {plink_data}

process maf {
publishDir params.publish
input:
set file (pl_files) from plink_data
output:
set file (test1) into result1
script:
"""
plink --bfile $pl_files --maf 0.05 --make-bed --out $test1
"""
}
process gen {
publishDir params.publish
input:
set file (test1) from result1
output:
set file (test2) into result2
script:
"""
plink --bfile $test1 --geno 0.05 --make-bed --out $test2
"""
}
process maf2 {
publishDir params.publish
input:
set file (test2) from result2
output:
set file (test3) into result3
script:
"""
plink --bfile $test2 --maf 0.01 --make-bed --out $test3
"""
}
`

[zillurbmb51]

Hi @abhi18av,
If I use test1 instead of $test1 I get
Error executing process > 'maf (1)' Caused by: Missing output file(s) 'test1' expected by process 'maf (1)'