Cromwell call caching and scatter

[ParkvilleData]

Hi,

I have a question/require help. When implementing call caching and using a scatter, if a task/shard fails does cromwell restart then entire task?

Example. I am running ~200 alignments creating ~200 bam files, rather large ones, say if 190 complete and the last one fails, when I restart I am seeing that the entire 200 bam alignments are run again. Is it possible to have cromwell's call caching resume after the 190 completed alignments as opposed to rerunning them?

Has anyone come across this? I am thinking it might be possible using the following parameters

1. • ContinueWhilePossible = true
2. • System.file-hash-cache = true
3. • System.graceful-server-shutdown = true

We are currently trying this but I wanted to see if anyone has come across this as well?

Thanks for your help!

[ParkvilleData]

Update:

Issue is that we were updating the resource requests and cromwell was seeing this as a new run, is there anyway to update the resource requests while allowing cromwell to resume the call-caching?

Thanks!

[ParkvilleData]

It seems like memory-retry may work for retrying with more memory, is there an option to increase the runtime?

Here is the adding of the memory-retry
#5180