-
Notifications
You must be signed in to change notification settings - Fork 235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Turn ObjectStoreLocationProvider
off by default
#1722
Turn ObjectStoreLocationProvider
off by default
#1722
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing this @kevinjqliu.
I think it would be good to add this to 0.9.0
reverts #1509, cc @smaheshwar-pltr |
Thanks for the ping @kevinjqliu. (Feel free to respond later) Curious what the problem here was. I'd have thought that old tables would still be readable by new PyIceberg and old PyIceberg would still be able to read tables created by the new one. Is changing locations just too dramatic for this release? @Fokko's #1452 (comment) that prompted this still seems reasonable (maybe longer-term), or is it not? |
My main concern was around the default behavior change. For example, if i had a pipeline running with 0.8.1 and now i upgrade to 0.9.0. The data file location is suddenly changed to use the hashed path. We can add it to the release notes, but i think changing the default behavior will surprise some users. |
Thanks @kevinjqliu for the explanation. |
:) Sorry for the back and forth! |
Closes #1721
Otherwise, there's a default behavior change in new 0.9.0 release. Previous versions will write to
data/
, new version will write todata/<hash>/