perf: Initially trim leading and trailing filtered rows #19850
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a small step before doing a `filter` and trims away the leading and
trailing filtered rows. This cost us practically nothing in the cases where
there are no items to trim and saves massively where it can trim items. This is
for example in cases where we are filtering by at least one sorted column.
Here is a code snippet to show the effectiveness. Before, it had to copy the
memory and go into the filter loop. Now, it can just take the fast path. This
also holds for sorted columns which is where I think most of the performance
will come from.
```python
import numpy as np
import polars as pl
import os
from timeit import timeit
SIZE = 10_000_000
os.environ['POLARS_MAX_THREADS'] = '1'
df = pl.DataFrame([
pl.Series('f', sum(([i] * (SIZE // 10) for i in range(10)), []), pl.Int32),
] + [
pl.Series(f'data{i}', np.random.randint(0, 500, SIZE), pl.Int64)
for i in range(100)
])
t = timeit(lambda: df.filter(pl.col.f == 5), number=10)
print(f"{t}s")
```
```console
Before: 1.1825743649969809s
After: 0.07909882599778939s
```
coastalwhite
requested review from
ritchie46,
orlp and
c-peters
as code owners
github-actions
bot
added
performance
|
Wow! Great idea. And basically free indeed. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #19850 +/- ##
==========================================
+ Coverage 79.37% 79.40% +0.02%
==========================================
Files 1550 1550
Lines 214743 214826 +83
Branches 2460 2447 -13
==========================================
+ Hits 170457 170575 +118
+ Misses 43727 43694 -33
+ Partials 559 557 -2 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
ritchie46
reviewed
Nov 18, 2024
| // is very minimal: only a clone of the mask and the array. | ||
| // | ||
| // This also allows dispatching to the fast paths way, way, way more often. | ||
| let mut mask = mask.clone(); |
There was a problem hiding this comment.
Maybe we should do this one line 51? After the full and empty check.
There was a problem hiding this comment.
No the effectiveness stems exactly from doing it before. If we trim to an exact full or empty block, we can take the fast path.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a small step before doing a
filterand trims away the leading and trailing filtered rows. This cost us practically nothing in the cases where there are no items to trim and saves massively where it can trim items. This is for example in cases where we are filtering by at least one sorted column.Here is a code snippet to show the effectiveness. Before, it had to copy the memory and go into the filter loop. Now, it can just take the fast path. This also holds for sorted columns which is where I think most of the performance will come from.