[CHORE] [ScanOperator-Follow-Ons-3] Integrate GlobScanOperator with new scan node builder #1564
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ew scan node builder
9 tasks
jaychia
added a commit
that referenced
this pull request
Nov 7, 2023
This PR adds an e2e integration for the new `ScanOperator` for reading from external sources, integrating with logical plan building, logical -> physical plan translation, physical plan scheduling, physical task execution, and the actual `MicroPartition`-based reading. ## TODOs (possibly before merging) - [ ] Implement Python I/O backend at `MicroPartition` level. - [ ] Implement reads for non-Parquet formats at `MicroPartition` level. - [x] Consolidate filter/limit pushdowns to use the same `Pushdown` struct. - [x] Look to reinstate non-optional `TableMetadata` at the `MicroPartition` level. (#1563) - [x] Look to reinstate non-optional `TableStatistics` when data is unloaded at the `MicroPartition` level. (#1563) - [x] Integrate with globbing `ScanOperator` implementation. (#1564) - [ ] Support different row group selection per Parquet file (currently applies a single row group selection to all files in a scan task bundle). - [ ] Misc. cleanup. - [ ] (?) Add basic validation that `ScanTask` configurations are compatible when merging into a `ScanTaskBatch` bundle. --------- Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com> Co-authored-by: Jay Chia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.