Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArrayDataget_slice_memory_size or similar #3407

Closed
alamb opened this issue Dec 29, 2022 · 4 comments · Fixed by #3501
Closed

ArrayDataget_slice_memory_size or similar #3407

alamb opened this issue Dec 29, 2022 · 4 comments · Fixed by #3501
Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@alamb
Copy link
Contributor

alamb commented Dec 29, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As part of implementing an Arrow flight client, we have observed that there is a maximum message size for gRPC messages, and thus while encoding the Arrow flight client has a way to limit the size of the messages (see #3391)

However, apparently there is currently no way to get the memory size used by a sliced (aka the array only points to a portion of the overall underlying allocation), which will result in overly conservative re-splitting in the arrow-flight client

Describe the solution you'd like
As suggested by @tustvold on #3391 (comment)

It is worth noting that this method does not take into account already sliced arrays, we likely need to add a get_slice_memory_size or something to ArrayData to provide this, it should be relatively straightforward

Describe alternatives you've considered
We could leave the users of the client to worry about batch splitting

Additional context

@alamb alamb added the enhancement Any new improvement worthy of a entry in the changelog label Dec 29, 2022
@askoa
Copy link
Contributor

askoa commented Jan 9, 2023

I'll give this a try.

@alamb
Copy link
Contributor Author

alamb commented Jan 9, 2023

Thank you @askoa

@tustvold tustvold added the arrow Changes to the arrow crate label Jan 13, 2023
@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow'} from #3501

@tustvold tustvold added the arrow-flight Changes to the arrow-flight crate label Jan 13, 2023
@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow-flight'} from #3391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants