MarkDuplicatesSpark improvements checkpoint#4656

Merged

droazen merged 27 commits intomasterfrom lb_modify_to_zip_partitions

Apr 26, 2018

+1,119-432

Commits on Apr 18, 2018

Changed the key generation in MarkDuplicates to use a StringBuilder
jamesemery
committed
Further Optimizations
jamesemery
committed
Do check on CompareDuplicatesSpark because its probably broken now
jamesemery
committed
adding in the registrator changes

lbergelson
authored and
jamesemery
committed
removing a sort and maybe making duplicating slightly more efficient
jamesemery
committed
A prototype that should demonstrate the theoretical cost of running the tool with smaller serializing
jamesemery
committed
making changes to mark based on zip partitions

lbergelson
authored and
jamesemery
committed
refactoring paired ends

lbergelson
authored and
jamesemery
committed
fixing wierd multimap issue

lbergelson
authored and
jamesemery
committed
making changes to fix a few things

lbergelson
authored and
jamesemery
committed
disabling non-query sorted input and encapsulating fields in PairedEnds

lbergelson
authored and
jamesemery
committed
working on custom serializers

lbergelson
authored and
jamesemery
committed
more work in this rat race that is optical duplicate marking
jamesemery
committed
finishing up custom partitioner round 1

lbergelson
authored and
jamesemery
committed
all mark duplicates test passing

lbergelson
authored and
jamesemery
committed
actually set the sort order...

lbergelson
authored and
jamesemery
committed
stopgap to fix temporary insanity
jamesemery
committed
adding some instrumentation to hopefully catch the bug and understand what is going on
jamesemery
committed
Refactored the pairedends code and added propper support for secondary and suppelementary reads
jamesemery
committed
Added an argument to disable duplciate marking of unmapped mates
jamesemery
committed
Fixed the broken kryo registration
jamesemery
committed

Commits on Apr 24, 2018

Responded to the first round of comments
jamesemery
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MarkDuplicatesSpark improvements checkpoint#4656

Commits on Apr 18, 2018

Changed the key generation in MarkDuplicates to use a StringBuilder

Further Optimizations

Do check on CompareDuplicatesSpark because its probably broken now

adding in the registrator changes

removing a sort and maybe making duplicating slightly more efficient

A prototype that should demonstrate the theoretical cost of running the tool with smaller serializing

making changes to mark based on zip partitions

refactoring paired ends

fixing wierd multimap issue

making changes to fix a few things

disabling non-query sorted input and encapsulating fields in PairedEnds

working on custom serializers

more work in this rat race that is optical duplicate marking

finishing up custom partitioner round 1

all mark duplicates test passing

actually set the sort order...

stopgap to fix temporary insanity

adding some instrumentation to hopefully catch the bug and understand what is going on

Refactored the pairedends code and added propper support for secondary and suppelementary reads

Added an argument to disable duplciate marking of unmapped mates

Fixed the broken kryo registration

Commits on Apr 24, 2018

Responded to the first round of comments

Commits on Apr 25, 2018

still unclear what happened

oh right, i needed that file didn't I?

fixing a silly copy error

someday bugs will be eradicated automatically by executing the code to first see if it works

responded to yet more comments