Performance of methylation pipeline

[naumenko-sa]

Users reported slowdowns of methylation pipeline on trim_galore and bismark steps.

1. trim_galore uses a non-straightforward threading scheme:

• https://github.com/FelixKrueger/TrimGalore/blob/master/Docs/Trim_Galore_User_Guide.md
• Multithreaded support from Cutadapt in trim_galore FelixKrueger/TrimGalore#16
• https://github.com/bcbio/bcbio-nextgen/blob/master/bcbio/wgbsseq/trimming.py#L44

solution: with bcbio_nextgen.py -n 4 trim_galore runs cutadapt with 1 thread and it is very slow. Increasing bcbio treads speeds up trim_galore.

2. We tested bismark step with two threading parameters:
   https://bcbio-nextgen.readthedocs.io/en/latest/contents/methylation.html#benchmarking
   The currently recommended combination is 16/2/100G.

#3303
#3301

S

[naumenko-sa]

Some samples failed extractor step:

bismark_methylation_extractor \
--no_overlap \
--comprehensive \
--cytosine_report \
--genome_folder /genomes/Hsapiens/hg38/bismark/ \
--merge_non_CpG \
--multicore 1 \
--buffer_size 5G \
--bedGraph \
--gzip 
/path/work/dedup/sample/sample.nsorted.deduplicated.bam

[FATAL ERROR:] The IDs of Read 1 and Read 2 are not the same.
This might be the result of sorting the BAM files by chromosomal position or merging several files with Samtools sort, and this is not compatible with correct methylation extraction. Please use an unsorted file instead or sort the file by name using the command 'samtools sort -n'. Paired-end files may be merged properly (without risking this error) using either 'samtools merge -n' or 'samtools cat'.

[naumenko-sa]

it seems to happen in some samples because we do

1. bismark alignment
2. sorting
3. deduplication
4. extraction

step 4 fails with sorted reads, but for 3 we have to sort. the solution is to skip sorting and deduplication or do step4 in a single-end mode (-s).

[naumenko-sa]

FelixKrueger/Bismark#360

[naumenko-sa]

Lambda phage discussion: FelixKrueger/Bismark#361

[naumenko-sa]

Finished the test cohort + vs Lambda genome.
4/2/100G (-n 16) passes without errors.
For the fastest processing use 16/2/192G (-n 32)