Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BitMaskedArray #58

Closed
jpivarski opened this issue Jan 14, 2020 · 2 comments · Fixed by #143
Closed

BitMaskedArray #58

jpivarski opened this issue Jan 14, 2020 · 2 comments · Fixed by #143
Assignees
Labels
feature New feature or request

Comments

@jpivarski
Copy link
Member

Ironically, IndexedMaskedArray (from FillableArray) and BitMaskedArray (from Arrow/Parquet) are more important than ByteMaskedArray. The ByteMaskedArray might actually lag behind and not get implemented for a while.

Might need to change the BitMaskedArray wiki page.

@jpivarski jpivarski self-assigned this Jan 14, 2020
@jpivarski jpivarski added the feature New feature or request label Jan 14, 2020
@jpivarski jpivarski added this to the Needed for Arrow/Parquet milestone Jan 14, 2020
@jpivarski
Copy link
Member Author

I should do ByteMaskedArray before BitMaskedArray, as preparation. BitMaskedArray is just like it but with the additional complication of having to deal with bits. Probably just generate bytes when needed, leaving direct bit-manipulation as a later optimization.

As soon as ByteMaskedArray is ready, reducers should get a semigroup parameter to turn placeholder-identities into None (expected for min/max, and probably necessary before #70).

@jpivarski
Copy link
Member Author

At that time, also push None values in a reduction down to just above the primitive level with ByteMaskedArray, and use that to propagate Nones in the result. But it will be much easier with a ByteMaskedArray than an IndexedOptionArray.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant