index out of range when encoding large VCF

[JakeHagen]

Hi Brent

Thank you for another great tool. I am getting this error when trying to encode a rather large VCF. The VCF is spliceAI indels, with a custom max field (max of four values). The json config is also below. I am also trying to attach the smallest VCF I can make that would still produce the error. (hopefully you can download this https://www.icloud.com/iclouddrive/0967utbVnXQf9DsBnTWh4tiqA#chr5 , sorry it is so large, if I drill down to smaller locations, the error goes away)

[
  {"field": "SpliceAI_max", "alias": "SpliceAI_max", "multiplier": 200},
  {"field": "SpliceAI_max_name", "alias": "SpliceAI_max_name"},
]

RUST_BACKTRACE=full echtvar encode spliceai_scores.masked.indel.hg38.max.chr5.zip spliceai_scores.masked.hg38.max.echtvar.json spliceai_scores.masked.indel.h
g38.max.chr5.vcf.gz
[echtvar] adding VCF:spliceai_scores.masked.indel.hg38.max.chr5.vcf.gz
thread 'main' panicked at 'range end index 39591728 out of range for slice of length 39591726', /home/brentp/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/stream-vbyte-0.4.0/src/encode/sse41.rs:117:21
stack backtrace:
   0:           0x4fc31d - std::backtrace_rs::backtrace::libunwind::trace::h65ef482bb9b15649
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:           0x4fc31d - std::backtrace_rs::backtrace::trace_unsynchronized::hf1ee7630128bf9a9
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:           0x4fc31d - std::sys_common::backtrace::_print_fmt::haddc20e8865333bd
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:67:5
   3:           0x4fc31d - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hc06b166f304d5a13
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:46:22
   4:           0x49cebc - core::fmt::write::h212f7b7266b9a26a
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/fmt/mod.rs:1149:17
   5:           0x4dae84 - std::io::Write::write_fmt::h240619c436cddbd8
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/io/mod.rs:1660:15
   6:           0x4fdbae - std::sys_common::backtrace::_print::h9d519f4e309ac3d8
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:49:5
   7:           0x4fdbae - std::sys_common::backtrace::print::h8362bac372870d46
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:36:9
   8:           0x4fdbae - std::panicking::default_hook::{{closure}}::hdf453592ec91e76e
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:211:50
   9:           0x4fdbae - std::panicking::default_hook::h9b626066b2c6b270
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:228:9
  10:           0x4fdbae - std::panicking::rust_panic_with_hook::h90c32e4f1cc54562
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:606:17
  11:           0x4fd468 - std::panicking::begin_panic_handler::{{closure}}::h6751672be4522935
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:502:13
  12:           0x4fd3e6 - std::sys_common::backtrace::__rust_end_short_backtrace::hdecab70784de07d3
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:139:18
  13:           0x4fd3a2 - rust_begin_unwind
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:498:5
  14:           0x404bd0 - core::panicking::panic_fmt::h3a6bf5e754065d1c
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/panicking.rs:107:14
  15:           0x404cd1 - core::slice::index::slice_end_index_len_fail::h8d2ca6c452b724f9
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/slice/index.rs:43:5
  16:           0x41689d - echtvar::commands::encoder_cmd::write_bits::hbb96f375a38dcffc
  17:           0x41aff5 - echtvar::commands::encoder_cmd::encoder_main::hbcd6dffce26afc25
  18:           0x425241 - echtvar::main::h994cda02b319af02
  19:           0x433863 - std::sys_common::backtrace::__rust_begin_short_backtrace::h3a3ce605638829d4
  20:           0x43376d - std::rt::lang_start::{{closure}}::h95dc532fc021057b
  21:           0x4345cb - main
  22:           0x78483a - __libc_start_call_main
  23:           0x785cc7 - __libc_start_main_impl
  24:           0x408b85 - _start
  25:                0x0 - <unknown>

[brentp]

Hi, thanks for reporting. Looks like this might be an issue with the stream-vbyte lib that I am using.
Can you try this debug binary? (gunzip, chmod+x and run).
It will also output position information that might help you to create a smaller test-case. I'll also have a look at your file on monday.

[brentp]

Actually, your file isn't so large. I'll take it from here.

[brentp]

Hi, I pushed a fix for this. Will you try this binary?
If it works for you, I'll make a new release in the coming days. Thanks again for reporting!

[JakeHagen]

Wow that was fast. That binary works for me on the small VCF I created. I am now testing it on the whole VCF, and should be able to report back in a couple days.
Thanks again for the quick fix and great tool.

[brentp]

Thanks for reporting! spliceAI is definitely a good stress test for echtvar.

I found another issue using the file you sent that's still present in that binary (so don't continue to use that one). I have just fixed that as well. I'll make a release next week.
echtvar.gz
You can try this one or wait for the release.

[JakeHagen]

Awesome, thanks again.

[brentp]

Just verifying this is resolved for you. Please let me know any other issues.
Thanks again

[JakeHagen]

Sorry for not giving an update, yes it was resolved. Thanks you

[brentp]

No problem. Just wanted to verify.