Zero-copy buffer protocol data import #204
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change list
AnyBufferProtocol
enum, with variants for all theElement
types. In theFromPyObject
impl onAnyBufferProtocol
, we simply check for each Element type.into_arrow_array
for zero-copy conversions of buffer protocol objects. This relies on theBuffer::from_custom_allocation
method inarrow
. This is safe because the originalPyBuffer
is put into anArc
and tracked by the Arrow buffer. So when the Arrow array is dropped, it will decrement the reference count on theArc
, and when it hits zero theArc<PyBuffer>
will be dropped. This will at that point call the original buffer's release method.FromPyObject
impl onPyArray
.PyArray
now also accept buffer protocol inputPyAnyBuffer
from Support for python buffer protocol #156. This was a lot of low-level, relatively unsafe code, that is not ideal to maintain here.ArrayInput
type union, which is a union ofArrowArrayExportable
and buffer protocol objects.ArrayInput
instead ofArrowArrayExportable
where appropriateWe're now able to do computations on numpy arrays by default!
This really isn't bad! Especially as numpy might be doing SIMD here (the arrow crate is SIMD capable, but we aren't enabling SIMD optimizations in our wheel builds).