Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid importing polars in arrow scan #4551

Merged
merged 1 commit into from
Nov 19, 2024
Merged

Avoid importing polars in arrow scan #4551

merged 1 commit into from
Nov 19, 2024

Conversation

acquamarin
Copy link
Collaborator

@acquamarin acquamarin commented Nov 19, 2024

This PR removes the dependency of polars library when scanning from arrow.

@acquamarin acquamarin requested a review from ray6080 November 19, 2024 17:00
Copy link

Benchmark Result

Master commit hash: 6f1bf5b60107419eb2f4d947ca0fd8ed477f72b0
Branch commit hash: c2d822ca6042c12d618128edfce7e7d466148300

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 643.89 642.95 0.95 (0.15%)
aggregation q28 11417.95 11469.90 -51.95 (-0.45%)
filter q14 124.92 126.43 -1.50 (-1.19%)
filter q15 125.04 123.37 1.66 (1.35%)
filter q16 302.19 303.19 -1.01 (-0.33%)
filter q17 446.81 444.82 1.99 (0.45%)
filter q18 1940.93 1946.70 -5.77 (-0.30%)
filter zonemap-node 86.96 86.57 0.39 (0.45%)
filter zonemap-node-lhs-cast 86.49 86.53 -0.04 (-0.05%)
filter zonemap-rel 5529.55 5743.59 -214.04 (-3.73%)
fixed_size_expr_evaluator q07 541.99 544.45 -2.46 (-0.45%)
fixed_size_expr_evaluator q08 757.94 778.80 -20.85 (-2.68%)
fixed_size_expr_evaluator q09 758.82 774.16 -15.34 (-1.98%)
fixed_size_expr_evaluator q10 237.87 238.02 -0.15 (-0.06%)
fixed_size_expr_evaluator q11 232.23 237.01 -4.79 (-2.02%)
fixed_size_expr_evaluator q12 233.57 236.82 -3.25 (-1.37%)
fixed_size_expr_evaluator q13 1458.36 1503.38 -45.02 (-2.99%)
fixed_size_seq_scan q23 115.43 119.91 -4.48 (-3.74%)
join q29 623.92 662.44 -38.52 (-5.81%)
join q30 1417.13 1289.55 127.58 (9.89%)
join q31 4.93 7.01 -2.08 (-29.65%)
ldbc_snb_ic q35 418.99 406.73 12.26 (3.02%)
ldbc_snb_ic q36 136.06 126.12 9.94 (7.89%)
ldbc_snb_is q32 5.50 5.16 0.34 (6.63%)
ldbc_snb_is q33 12.73 15.12 -2.39 (-15.80%)
ldbc_snb_is q34 1.25 1.61 -0.36 (-22.49%)
multi-rel multi-rel-large-scan 1599.76 1612.54 -12.78 (-0.79%)
multi-rel multi-rel-lookup 5.44 21.53 -16.09 (-74.73%)
multi-rel multi-rel-small-scan 96.93 99.58 -2.65 (-2.66%)
order_by q25 132.50 131.52 0.98 (0.75%)
order_by q26 452.63 460.60 -7.97 (-1.73%)
order_by q27 1415.30 1435.41 -20.10 (-1.40%)
scan_after_filter q01 169.34 174.04 -4.70 (-2.70%)
scan_after_filter q02 156.18 157.98 -1.80 (-1.14%)
shortest_path_ldbc100 q37 79.58 84.91 -5.33 (-6.28%)
shortest_path_ldbc100 q38 472.68 418.99 53.70 (12.82%)
shortest_path_ldbc100 q39 63.36 63.00 0.36 (0.58%)
shortest_path_ldbc100 q40 527.83 514.13 13.70 (2.67%)
var_size_expr_evaluator q03 2070.07 2102.75 -32.68 (-1.55%)
var_size_expr_evaluator q04 2245.48 2268.57 -23.09 (-1.02%)
var_size_expr_evaluator q05 2694.62 2699.13 -4.51 (-0.17%)
var_size_expr_evaluator q06 1346.82 1345.60 1.22 (0.09%)
var_size_seq_scan q19 1475.79 1479.78 -3.99 (-0.27%)
var_size_seq_scan q20 2559.08 2767.61 -208.53 (-7.53%)
var_size_seq_scan q21 2301.91 2317.91 -15.99 (-0.69%)
var_size_seq_scan q22 131.32 130.39 0.93 (0.72%)

Copy link

codecov bot commented Nov 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.37%. Comparing base (9d5afd1) to head (aee5e60).
Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4551      +/-   ##
==========================================
- Coverage   87.37%   87.37%   -0.01%     
==========================================
  Files        1349     1349              
  Lines       56622    56622              
  Branches     7131     7130       -1     
==========================================
- Hits        49474    49471       -3     
- Misses       6976     6979       +3     
  Partials      172      172              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@acquamarin acquamarin merged commit de8edf9 into master Nov 19, 2024
25 checks passed
@acquamarin acquamarin deleted the avoid-polars branch November 19, 2024 21:01
ray6080 pushed a commit that referenced this pull request Dec 17, 2024
ray6080 pushed a commit that referenced this pull request Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants