-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Raise proper error for mismatching parquet schema instead of panicking #17321
Conversation
Hi, just a quick note that in the first post you need to have one "fixes" per issue number to close them, e.g. "fixes #A, #B, #C" will only close #A :) |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #17321 +/- ##
=======================================
Coverage 80.72% 80.73%
=======================================
Files 1475 1475
Lines 193333 193383 +50
Branches 2760 2760
=======================================
+ Hits 156077 156119 +42
- Misses 36746 36754 +8
Partials 510 510 ☔ View full report in Codecov by Sentry. |
Very cool tip, thanks 🙂 |
@nameexhaustion @ritchie46, but this doesn't add support read parquets with mixed schema's though. This is something pyarrow supports |
Yeah we should be careful not to close issues when a panic has been turned into an error, if it should not throw an error in the first place. In this case, mismatching schemas indicate schema evolution and we may have to actually support this. |
This has been reported quite a bit lately, so got around to it
Fixes #13436, fixes #17067, fixes #17255, fixes #13438
Maybe also #17311, but I don't have a repro for that one