-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GH-43202: [C++][Compute] Detect and explicit error for offset overflow in row table #43226
Conversation
|
Hi @pitrou @westonpace , would you like to take a look? Thanks. |
@github-actions crossbow submit -g cpp |
This comment was marked as outdated.
This comment was marked as outdated.
@github-actions crossbow submit -g cpp |
Revision: ae34ac4 Submitted crossbow builds: ursacomputing/crossbow @ actions-72e104db5c |
{ | ||
row_encoder.PrepareEncodeSelected(0, num_rows - 1, columns); | ||
ASSERT_OK(row_encoder.EncodeSelected(&row_table, static_cast<uint32_t>(num_rows - 1), | ||
row_ids.data())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we drop/clear the row_table
after this to release a bit of memory?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for looking!
Every time an encoder encodes into a row table, the row table gets effectively cleared. So the memory is only hold til the next EncodeSelected
call. Clearing it here doesn't make much difference I guess(?) - the code in between is relatively trivial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, ok, thank you
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit cf5d7c7. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them. |
Rationale for this change
There are two places for the offset in a row table to possibly overflow: 1) Encoding columns into a row table; 2) Appending to a row table from another row table.
They are particularly bad because they can cause silent wrong result for the computation.
What changes are included in this PR?
Detect such overflows in the aforementioned places and report an explicit error when an overflow is detected.
Are these changes tested?
UT included.
Are there any user-facing changes?
User code that used to be fake passing could see an explicit error after this change.
This PR contains a "Critical Fix".