Test failures with bowtie2 2.3.1

[satta]

Some tests in Ariba 2.9.0 fail when using the latest Bowtie2 version (2.3.1):

======================================================================
FAIL: Test run_bowtie2 unsorted
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/ariba-2.9.0+ds/ariba/tests/mapping_test.py", line 61, in test_run_bowtie2
    self.assertListEqual(expected, got)
AssertionError: Lists differ: [('1', 99, 'ref', 0, [(4, 5), (0, 20)], 'AGCCCTC[857 chars]AT')] != [('1', 153, 'ref', 30, [(0, 25)], 'AGGATACAGATCT[794 chars]AT')]

First differing element 0:
('1', 99, 'ref', 0, [(4, 5), (0, 20)], 'AGCCCTCCACAGGATGGTGGTATAC')
('1', 153, 'ref', 30, [(0, 25)], 'AGGATACAGATCTTGTGGGAAAGGT')

- [('1', 99, 'ref', 0, [(4, 5), (0, 20)], 'AGCCCTCCACAGGATGGTGGTATAC'),
-  ('1', 147, 'ref', 30, [(0, 25)], 'AGGATACAGATCTTGTGGGAAAGGT'),
? ^       ^^

+ [('1', 153, 'ref', 30, [(0, 25)], 'AGGATACAGATCTTGTGGGAAAGGT'),
? ^       ^^

+  ('1', 69, None, 30, [], 'AGCCCTCCACAGGATGGTGGTATAC'),
-  ('2', 99, 'ref', 124, [(0, 25)], 'TAATGTTCTTAGGGCTTACCATAGA'),
?        ^^

+  ('2', 73, 'ref', 124, [(0, 25)], 'TAATGTTCTTAGGGCTTACCATAGA'),
?        ^^

-  ('2', 147, 'ref', 170, [(0, 20), (4, 5)], 'TCCACCTTAGCTAAGCGCAGACTCG'),
+  ('2', 133, None, 124, [], 'CGAGTCTGCGCTTAGCTAAGGTGGA'),
   ('3', 73, 'ref', 86, [(0, 25)], 'TCGGGTCTGTACAAGGACGGATGGT'),
   ('3', 133, None, 86, [], 'CGTACTGACTGACTGACGTACTGCA'),
   ('4', 99, 'ref', 55, [(0, 25)], 'CCGCCGGGAAGTCCTTCTGTCGTGC'),
   ('4', 147, 'ref', 136, [(0, 25)], 'GGCTTACCATAGAGGTACACTAAAA'),
-  ('5', 99, 'ref', 0, [(4, 2), (0, 23)], 'CCTCCACAGGATGGTGGTATACCTG'),
?        ^^  ^^ --  ^   ---------------

+  ('5', 77, None, -1, [], 'CCTCCACAGGATGGTGGTATACCTG'),
?        ^^  ^^^   ^^

-  ('5', 147, 'ref', 166, [(0, 24), (4, 1)], 'TTCATCCACCTTAGCTAAGCGCAGA'),
+  ('5', 141, None, -1, [], 'TCTGCGCTTAGCTAAGGTGGATGAA'),
   ('6', 77, None, -1, [], 'CAGTTGCATGACGTCATGCAGTCAT'),
   ('6', 141, None, -1, [], 'AATGAGTATGATGAGTAATGGTATG'),
   ('7', 99, 'ref', 56, [(4, 1), (0, 23), (4, 1)], 'ACGCCGGGAAGTCCTTCTGTCGTGT'),
   ('7', 147, 'ref', 136, [(0, 24), (4, 1)], 'GGCTTACCATAGAGGTACACTAAAT')]

======================================================================
FAIL: Test run_bowtie2 sorted
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/ariba-2.9.0+ds/ariba/tests/mapping_test.py", line 102, in test_run_bowtie2_and_sort
    self.assertListEqual(expected, got)
AssertionError: Lists differ: [('1', 99, 'ref', 0, [(4, 5), (0, 20)], 'AGCCCTC[857 chars]TG')] != [('1', 69, None, 30, [], 'AGCCCTCCACAGGATGGTGGTA[794 chars]TG')]

First differing element 0:
('1', 99, 'ref', 0, [(4, 5), (0, 20)], 'AGCCCTCCACAGGATGGTGGTATAC')
('1', 69, None, 30, [], 'AGCCCTCCACAGGATGGTGGTATAC')

Diff is 1432 characters long. Set self.maxDiff to None to see it.

----------------------------------------------------------------------

it looks like some results do not match the expected reference. It might have to do with the fact that in the latest version:

The default ---score-min for --local mode is now 'G,20,8'. That was the stated default in the documentation for a while, but the actual default was 'G,0,10' for many versions. Now the default matches the documentation and, we find, yields more accurate alignments than 'G,0,10'

It would be nice if the behaviour of Ariba when given various Bowtie2 versions could be made consistent as it currently causes the tests to fail, breaking builds. We have, for example, run into this issue on Debian since Debian's update to the latest Bowtie version.

[martinghunt]

Thanks Sascha, I'll look into it. At the moment it uses --very-sensitive-local. I'll see if I can change that to explicitly listing all the options that --very-sensitive-local changes, so then should be consistent across bowtie2 versions.

[satta]

Thanks a lot Martin! :)

[martinghunt]

This is going to need more thought. I presume having a specific version of bowtie2 is not really an option for debian packaging?

The combination of options for bowtie 2.2.9 that makes the ariba tests pass is impossible to use with bowtie 2.3.1 because bowtie2 throws an 'internal bowtie2 exception' about inconsistent options.

[satta]

This is going to need more thought. I presume having a specific version of bowtie2 is not really an option for debian packaging?

Unfortunately not. Maybe it would already help adjusting the test expectations if you're happy with the 2.3.1 results as well? No idea if the results are that much worse...

[martinghunt]

Should have said, results look worse with 2.3.1! Reporting a read as unmapped because it just needs to soft clip 1 base off the end. Maybe I'm missing another option. Will keep trying.

[satta]

Thanks for the update! Just let me know when you have decided on an option. I can imagine others will run into the same problem as well at some point...

[martinghunt]

Issue filed with bowtie2: BenLangmead/bowtie2#93

[satta]

It doesn't look like there's much progress and I'd like to find some solution to make sure Ariba stays in stretch. If all else fails do you think -- even though the results aren't perfect -- it would be OK to skip the problematic test to at least make sure the build completes?

[martinghunt]

Thanks Sascha. Might as well be pragmatic and skip it the test for now.

[martinghunt]

New release 2.9.2 has the fix.

[satta]

Great, thanks!