feat: vector search with distance range #3326
Conversation
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3326 +/- ##
==========================================
- Coverage 79.01% 78.97% -0.04%
==========================================
Files 246 246
Lines 87628 88095 +467
Branches 87628 88095 +467
==========================================
+ Hits 69238 69572 +334
- Misses 15523 15631 +108
- Partials 2867 2892 +25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| @@ -88,6 +94,8 @@ impl IvfSubIndex for FlatIndex { | |||
| dist: OrderedFloat(dist), | |||
| }) | |||
| .sorted_unstable() | |||
| .skip_while(|r| params.lower_bound.map_or(false, |lb| r.dist.0 < lb)) | |||
There was a problem hiding this comment.
should we do skip / take before sort?
Also giving most the case, low/high bounds are usually not set. Should we only run those two steps if low/high bounds are set, each of them is a O(n) operation.
There was a problem hiding this comment.
filter before sort would evaluate the prediction on all results then it's O(n) as well.
skip_while and take_while is lazy, so they would stop once take gets enough k results.
we can optimize this by binary search.
There was a problem hiding this comment.
There are two cases:
-
without lower_bound. /upper bound,
sorted_unstableis O(nlogn), where n = num of rows in each partition. -
with
lower_bound / upper bound,sorted_unstablecomplexity becomesO(n_2 * log(n_2))wheren_2is the filtered results?
The N in time complexity might be significantly different.
There was a problem hiding this comment.
Non the less, can we just run some benchmarks?
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
|
No description provided.