AssemblyEvidenceSource hangs

[mflevine]

Hi,

I am using gridss 2.2.0 in a container and gridss keeps hanging at AssemblyEvidenceSource

INFO	2019-02-27 05:55:20	SAMFileUtil	Sorting I-H-116984-T1-1-D1-1_2/somatic.gridss.assembly.bam.gridss.working/gridss.tmp.somatic.gridss.assembly.bam.assembly.chunk110.bam
INFO	2019-02-27 05:55:21	AssemblyEvidenceSource	Starting assembly on chunk 111 (6:46208041-6:56208040)

I am using config:

variantcalling.callUnassembledBreakends=false
variantcalling.callUnassembledBreakpoints=false

[d-cameron]

Can you post the output of a few calls to jstack [-l] <pid> (where <pid> is the process id of the java process running gridss) so I have an idea of where it is hanging? One likely cause is a assembly stalling in a relatively high coverage, low complexity region.

Configuration settings that can be lower to reduce the maximum assembly runtime are:

maxCoverage = 25000
assembly.downsample.targetEvidenceDensity = 6.0

The other most likely cause is insufficient memory. Run the jvisualvm or visualvm program that comes with your version of java, connect to the gridss process and looking at the time spent in garbage collection. If it's over 1%, you've run out of memory. Unfortunately, java programs that run out of memory hang for a very long time performing garbage collection before they actually get killed.

[d-cameron]

Did you set the WORKER_THREADS command line argument? GRIDSS default to one thread per core so if you're running the example script on a high-CPU count machine then you're very likely to run out of memory. In this case explicitly setting WORKER_THREADS and heap memory (-Xmx) to reasonable values (e.g `-Xmx31g WORKER_THREADS=16) should fix any out of memory hang for typical WGS sequencing data.

[mflevine]

I used 16 cores, -Xmx48g, and WORKER_THREADS=16, so I doubt its a memory issue. This sample also worked on versions 1.5.1 and 2.1.0.

I have attached the jstack output
gridss.hang.txt

[d-cameron]

It looks like it could be an edge case with the new multi-threaded BAM parsing code that was added in 2.2. If you remove -Dsamjdk.use_async_io_read_samtools=true from the command line (or replace it with -Dsamjdk.use_async_io_read_samtools=false, does this stop the hang?

[mflevine]

-Dsamjdk.use_async_io_read_samtools=false stopped the hang.

[iprada]

I had the same problem on various steps (variant calling and assembly) and
-Dsamjdk.use_async_io_read_samtools=false also solved my problems

[d-cameron]

Can you confirm v2.2.1 now fixes the problem for you input data?

[iprada]

Hi, at this moment I have many jobs that take as input the gridss file. I will be able to test this around friday

Cheers,

[mflevine]

Hi v2.2.1 does not hang but with -Dsamjdk.use_async_io_read_samtools=true the run took significantly longer. This time it took 72hrs on 16 cores. The merge steps seemed to take quite a long time.

[d-cameron]

Thanks for the feedback. I'll continue looking into it.

[d-cameron]

Should be fixed with v2.2.2.