Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support hive partitioning in scan_ipc #14885

Closed
claudious96 opened this issue Mar 6, 2024 · 0 comments · Fixed by #17434
Closed

Support hive partitioning in scan_ipc #14885

claudious96 opened this issue Mar 6, 2024 · 0 comments · Fixed by #17434
Assignees
@nameexhaustion
Labels
A-io-partitioning Area: reading/writing (Hive) partitioned files accepted Ready for implementation enhancement New feature or an improvement of an existing feature P-goal Priority: aligns with long-term Polars goals

Comments

@claudious96
Copy link

claudious96 commented Mar 6, 2024 •
edited
Loading

Description

It would be great to be able to read hive partitioned dataset in arrow's IPC format, natively in polars.

Right now it's still possible to achieve the scan by creating a pyarrow dataset and using pl.scan_pyarrow_dataset(ds), but it would be nice to have scan_ipc understand hive partitioning as it is done with scan_parquet.
@claudious96 claudious96 added the enhancement New feature or an improvement of an existing feature label Mar 6, 2024
@stinodego stinodego added the accepted Ready for implementation label Mar 29, 2024
@github-project-automation github-project-automation bot moved this to Ready in Backlog Mar 29, 2024
@stinodego stinodego added A-io-partitioning Area: reading/writing (Hive) partitioned files P-goal Priority: aligns with long-term Polars goals labels Mar 29, 2024
@stinodego stinodego mentioned this issue Apr 2, 2024
Closed
13 tasks
@nameexhaustion nameexhaustion self-assigned this Jul 5, 2024
@nameexhaustion nameexhaustion mentioned this issue Jul 5, 2024
Merged
@github-project-automation github-project-automation bot moved this from Ready to Done in Backlog Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nameexhaustion nameexhaustion

Labels
A-io-partitioning Area: reading/writing (Hive) partitioned files accepted Ready for implementation enhancement New feature or an improvement of an existing feature P-goal Priority: aligns with long-term Polars goals
Projects
Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Support hive partitioning in scan_ipc nameexhaustion/polars
3 participants
@stinodego @claudious96 @nameexhaustion