-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: Fix parquet file metadata is dropped after first DSL->IR conversion #18789
Conversation
/// Local use cases often repeatedly collect the same `LazyFrame` (e.g. in interactive notebook use-cases), | ||
/// so we cache the IR conversion here, as the path expansion can be quite slow (especially for cloud paths). | ||
#[cfg_attr(feature = "serde", serde(skip))] | ||
cached_ir: Arc<Mutex<Option<IR>>>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We had mutexes on multiple fields on the DSL::Scan
, I just cache the IR here instead of adding another mutex
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #18789 +/- ##
==========================================
- Coverage 79.84% 79.84% -0.01%
==========================================
Files 1518 1518
Lines 205563 205577 +14
Branches 2892 2892
==========================================
+ Hits 164137 164143 +6
- Misses 40878 40886 +8
Partials 548 548 ☔ View full report in Codecov by Sentry. |
Can we add a test somehow? |
4829c72
to
f7c0997
Compare
1 special order coming up 😁 |
CodSpeed Performance ReportMerging #18789 will not alter performanceComparing Summary
|
96b73ab
to
b52e5da
Compare
56fafe3
to
f996d44
Compare
This might've fixed the post 1.1.0 regression that was introduced somehow in the |
No description provided.