Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Puffin file format #744

Open
4 of 8 tasks
fqaiser94 opened this issue Nov 30, 2024 · 3 comments · May be fixed by #714
Open
4 of 8 tasks

Support Puffin file format #744

fqaiser94 opened this issue Nov 30, 2024 · 3 comments · May be fixed by #714
Assignees
@fqaiser94

Comments

@fqaiser94
Copy link
Contributor

fqaiser94 commented Nov 30, 2024 •
edited
Loading

Why

There has been increased interest in Iceberg's Puffin file format recently. This is partially driven by the fact that the Iceberg V3 Spec added support for deletion vectors which are expected to be stored in Puffin files.

However, as was recently noted on the dev-list, currently only the iceberg-java SDK supports reading or writing Puffin files. Iceberg-rust itself has zero support for the Puffin file format today. The purpose of this ticket is to change that by adding support for Puffin to iceberg-rust and iceberg-python (through exposed bindings).

How

I have already raised a PR for adding Puffin support to iceberg-rust here: #714. However as that PR is quite large, I am splitting up the code and submitting it as multiple PRs in the following order:
This was referenced Nov 30, 2024
Merged
Draft
@fqaiser94
Copy link
Contributor Author

Can someone assign this ticket to me to avoid confusion?

This was referenced Dec 7, 2024
Merged
Merged
@fqaiser94 fqaiser94 changed the title Iceberg-rust Puffin support Support Puffin file format Jan 15, 2025
@fqaiser94 fqaiser94 mentioned this issue Jan 15, 2025
Merged
@waynexia
Copy link
Member

We have also implemented a puffin inside greptimedb (https://github.com/GreptimeTeam/greptimedb/tree/main/src/puffin), would like to push this forward together 🍻

@liurenjie1024
Copy link
Contributor

We have also implemented a puffin inside greptimedb (https://github.com/GreptimeTeam/greptimedb/tree/main/src/puffin), would like to push this forward together 🍻

Great! Looking forward to see your contribution!

Xuanwo pushed a commit that referenced this issue Feb 8, 2025
@fqaiser94
feat(puffin): Add PuffinReader (#892)
a22440e 
Part of #744

# Summary 
- Add PuffinReader

# Context
- This is the fourth of a number of PRs to add support for Iceberg
Puffin file format.
- It might be helpful to refer to the overarching
[PR](#714) from which these
changes were split to understand better how these changes will fit in to
the larger picture.
- It may also be helpful to refer to the Java reference implementation
for PuffinReader
[here](https://github.com/apache/iceberg/blob/8cd5b1985d3f9c55ab2ced174559a8416b6ca1b4/core/src/main/java/org/apache/iceberg/puffin/PuffinReader.java#L123).
@fqaiser94 fqaiser94 mentioned this issue Feb 9, 2025
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqaiser94 fqaiser94

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(puffin): Add reader and writer fqaiser94/iceberg-rust
3 participants
@liurenjie1024 @waynexia @fqaiser94