Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix performance regression introduced during reader refactor #3170

Merged
merged 1 commit into from
Nov 25, 2024

Conversation

westonpace
Copy link
Contributor

In #2913 we added an initialization routine to the scheduler. Unfortunately, this caused a problem. It put schedule_ranges behind an await which means that we wouldn't schedule ranges for all files in a scan immediately. Instead, we wouldn't begin scheduling file X until we had reached nearly the last batch of file X-1. This introduced stuttering in the read path and had a significant effect on performance.

The fix here moves the initialization into the dedicated scheduler thread and keeps the creation of the scheduler a synchronous task which is actually a bit simpler anyways I think.

@github-actions github-actions bot added the bug Something isn't working label Nov 25, 2024
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 78.26087% with 5 lines in your changes missing coverage. Please review.

Project coverage is 78.02%. Comparing base (d79e870) to head (4be0c48).
Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
rust/lance-encoding/src/decoder.rs 78.26% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3170      +/-   ##
==========================================
+ Coverage   77.95%   78.02%   +0.07%     
==========================================
  Files         242      243       +1     
  Lines       82436    82890     +454     
  Branches    82436    82890     +454     
==========================================
+ Hits        64262    64676     +414     
- Misses      14960    15008      +48     
+ Partials     3214     3206       -8     
Flag Coverage Δ
unittests 78.02% <78.26%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@westonpace
Copy link
Contributor Author

Regression for this will be provided by #3165

@westonpace westonpace merged commit d196ab8 into lancedb:main Nov 25, 2024
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants