[CHORE] [ScanOperator-Follow-Ons-2] Refactor MicroPartition to have non-optional TableMetadata #1563
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
…on-optional TableMetadata
jaychia
force-pushed
the
jay/scan-operator-integration-metadata
branch
from
5389eca to
2a9ba32
Compare
jaychia
added a commit
that referenced
this pull request
Nov 7, 2023
This PR adds an e2e integration for the new `ScanOperator` for reading from external sources, integrating with logical plan building, logical -> physical plan translation, physical plan scheduling, physical task execution, and the actual `MicroPartition`-based reading. ## TODOs (possibly before merging) - [ ] Implement Python I/O backend at `MicroPartition` level. - [ ] Implement reads for non-Parquet formats at `MicroPartition` level. - [x] Consolidate filter/limit pushdowns to use the same `Pushdown` struct. - [x] Look to reinstate non-optional `TableMetadata` at the `MicroPartition` level. (#1563) - [x] Look to reinstate non-optional `TableStatistics` when data is unloaded at the `MicroPartition` level. (#1563) - [x] Integrate with globbing `ScanOperator` implementation. (#1564) - [ ] Support different row group selection per Parquet file (currently applies a single row group selection to all files in a scan task bundle). - [ ] Misc. cleanup. - [ ] (?) Add basic validation that `ScanTask` configurations are compatible when merging into a `ScanTaskBatch` bundle. --------- Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com> Co-authored-by: Jay Chia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Also refactors
MicroPartition::newinto explicit::new_loadedand::new_unloadedvariants:TableStatisticsisSomewhen the state isTableState::Unloadedat construction-timeTableStateno longer needs to be exposed externally and we can hide the calculations for TableMetadata inside of the::new_loadedconstructor::from_scan_task_batchis very explicit and clean. It tries::new_unloadedfirst, but falls back on::new_loadedif the ScanTaskBatch is not provided with both metadata/statistics