feat: add support for "take" operation to balanced storage #3079
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
westonpace
force-pushed
the
feat/take-blob-col
branch
2 times, most recently
from
afeb3f5 to
5285b86
Compare
westonpace
force-pushed
the
feat/take-blob-col
branch
from
5285b86 to
27974c6
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3079 +/- ##
==========================================
- Coverage 77.31% 77.16% -0.15%
==========================================
Files 240 240
Lines 79322 79561 +239
Branches 79322 79561 +239
==========================================
+ Hits 61326 61393 +67
- Misses 14821 15014 +193
+ Partials 3175 3154 -21
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
wjones127
approved these changes
Nov 5, 2024
Comment on lines
+121
to
+125
| # TODO: Once https://github.com/lancedb/lance/pull/3041 merges we will | ||
| # want to test partial appends. We need to make sure an append of | ||
| # non-blob data is supported. In order to do this we need to make | ||
| # sure a blob tx is created that marks the row ids as used so that | ||
| # the two row id sequences stay in sync. |
There was a problem hiding this comment.
👍 Thanks for testing the interaction
Comment on lines
+158
to
+161
| /// Splits the schema into two schemas, one with default storage class fields and the other with blob storage class fields. | ||
| /// If there are no blob storage class fields, the second schema will be `None`. | ||
| /// The order of fields is preserved. | ||
| pub fn partition_by_storage_class(&self) -> (Self, Option<Self>) { |
There was a problem hiding this comment.
Can only top-level fields be StorageClass::Blob?
There was a problem hiding this comment.
I think that is simplest. I should probably add that restriction. We can always add support for sub-fields in the future.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for "take" to datasets that have balanced storage
Requires balanced storage datasets to use stable ids
Refactors
RowAddressslightly to removeidfrom the method names (e.g.row_id->row_offset) to help avoid future confusion.Fixes an intermittent panic in
SharedStream(this panic had been seen before but now that we are usingSharedStreammore it was showing up more frequently)Begins aligning on new terminology for balanced storage (the old "blob storage" terminology is confusing because other things are also called "blobs":
Closes #3030