-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix infinite loop in not fully packed bit-packed runs #1555
Conversation
fyi @anliakho2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I am not super familiar with the code, I did review the test and verified it fails without this code change.
Also the code change makes sense to me.
Also I looked for other places that get_batch
is called. I found one in the DeltaBitPackedEncoder that maybe has a similar problem:
It also appears to be used a few other times.
/Users/alamb/Software/arrow-rs/parquet/src/column/reader/decoder.rs
260: Ok(reader.get_batch::<i16>(&mut out[range], *bit_width as usize))
/Users/alamb/Software/arrow-rs/parquet/src/data_type.rs
696: let values_read = bit_reader.get_batch(&mut buffer[..num_values], 1);
/Users/alamb/Software/arrow-rs/parquet/src/encodings/decoding.rs
655: .get_batch(&mut buffer[read..read + batch_to_read], bit_width);
/Users/alamb/Software/arrow-rs/parquet/src/encodings/levels.rs
271: decoder.get_batch::<i16>(&mut buffer[..len], bit_width as usize);
/Users/alamb/Software/arrow-rs/parquet/src/encodings/rle.rs
418: num_values = bit_reader.get_batch::<T>(
471: num_values = bit_reader.get_batch::<i32>(
/Users/alamb/Software/arrow-rs/parquet/src/util/bit_util.rs
1062: let values_read = reader.get_batch::<T>(&mut batch, num_bits);
@@ -743,6 +753,42 @@ mod tests { | |||
} | |||
} | |||
|
|||
#[test] | |||
fn test_truncated_rle() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW this test times out without the code in this PR 👍 which is a good sign to me that it covers the issue
test encodings::rle::tests::test_truncated_rle has been running for over 60 seconds
(I also tried removing each of the two cases -- dict and non dict and the test hung in both)
In this case the specification is very clear that the miniblock can't be truncated
And we return an error a line below if it is, so I think we should be ok. Good shout to check though 👍 The other cases don't appear to have loops, and GenericColumnReader which drives them will bail out if a page is truncated |
I plan to merge this after all (non miri) tests pass |
Which issue does this PR close?
Closes #1458
Cherry-picked from #1460, will add test and then mark ready for review
Rationale for this change
See ticket
What changes are included in this PR?
See ticket
Are there any user-facing changes?
No