Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-8616] [VL] Make filescan limit for encrypted fallback as configurable #8621

Merged
merged 1 commit into from
Feb 7, 2025

Conversation

ArnavBalyan
Copy link
Contributor

What changes were proposed in this pull request?

  • Making parquet file scan limit as configurable.
  • Currently it's hardcoded to default 10, now available as a user config can be changed externally.
  • Default remains the same as before of 10.

How was this patch tested?

  • Existing UTs

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Jan 26, 2025
Copy link

#8616

Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from acbcc3f to fc66b86 Compare January 26, 2025 09:48
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from fc66b86 to ce47d23 Compare January 26, 2025 12:06
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Contributor Author

cc @zhztheplayer could you please review thanks!

.doc("If supplied, `limit` number of files will be checked to determine encryption " +
"and falling back java scan")
.intConf
.createWithDefault(10)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: checkValue is required here to ensure the data is non-negative.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ArnavBalyan ArnavBalyan force-pushed the arnavb/configurable-limit branch from ce47d23 to 9a27d24 Compare January 28, 2025 17:02
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Contributor Author

cc @jackylee-ch fixed, can you please take a look

@Yohahaha Yohahaha merged commit 91800b4 into apache:main Feb 7, 2025
48 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_02_07_2025_time.csv log/native_master_02_06_2025_9aa21250fc_time.csv difference percentage
q1 16.34 16.43 0.092 100.56%
q2 16.42 14.47 -1.956 88.09%
q3 5.08 2.83 -2.254 55.66%
q4 86.63 85.55 -1.075 98.76%
q5 13.26 11.71 -1.545 88.35%
q6 5.39 3.72 -1.672 68.98%
q7 6.70 6.36 -0.345 94.85%
q8 4.66 6.62 1.968 142.28%
q9 28.40 29.47 1.070 103.77%
q10 13.29 12.20 -1.086 91.82%
q11 43.80 42.99 -0.806 98.16%
q12 2.31 2.13 -0.184 92.06%
q13 10.43 10.07 -0.365 96.50%
q14a 73.57 67.05 -6.519 91.14%
q14b 58.30 57.34 -0.960 98.35%
q15 3.31 3.73 0.423 112.79%
q16 29.50 29.96 0.466 101.58%
q17 8.49 7.37 -1.127 86.73%
q18 9.90 10.08 0.181 101.83%
q19 3.88 3.46 -0.419 89.19%
q20 2.34 4.35 2.007 185.70%
q21 1.54 1.50 -0.049 96.82%
q22 9.47 9.65 0.177 101.86%
q23a 137.99 136.87 -1.120 99.19%
q23b 163.33 162.14 -1.195 99.27%
q24a 109.31 101.26 -8.048 92.64%
q24b 92.81 96.16 3.352 103.61%
q25 6.58 6.32 -0.256 96.11%
q26 4.26 4.35 0.090 102.12%
q27 5.25 4.88 -0.377 92.82%
q28 35.57 38.90 3.329 109.36%
q29 18.06 18.64 0.576 103.19%
q30 8.74 6.85 -1.893 78.34%
q31 10.44 10.30 -0.142 98.64%
q32 2.62 2.01 -0.612 76.67%
q33 7.54 7.36 -0.179 97.63%
q34 4.65 4.67 0.020 100.43%
q35 11.38 10.69 -0.695 93.90%
q36 5.63 5.92 0.293 105.21%
q37 5.25 6.66 1.404 126.73%
q38 22.94 17.29 -5.646 75.38%
q39a 4.42 4.56 0.135 103.05%
q39b 5.12 5.16 0.045 100.87%
q40 6.57 5.35 -1.221 81.42%
q41 1.29 1.19 -0.101 92.19%
q42 1.18 1.19 0.012 101.04%
q43 4.57 4.49 -0.086 98.13%
q44 14.28 12.86 -1.418 90.07%
q45 4.52 4.74 0.224 104.95%
q46 5.79 5.51 -0.280 95.16%
q47 20.07 20.22 0.155 100.77%
q48 6.55 6.39 -0.161 97.55%
q49 10.12 10.35 0.230 102.27%
q50 39.28 38.54 -0.737 98.12%
q51 14.64 14.20 -0.436 97.02%
q52 1.16 1.48 0.327 128.24%
q53 3.03 2.86 -0.174 94.26%
q54 7.11 6.55 -0.566 92.05%
q55 1.39 3.51 2.118 252.51%
q56 6.95 7.43 0.485 106.98%
q57 13.69 13.09 -0.602 95.61%
q58 3.40 3.33 -0.070 97.94%
q59 6.83 6.63 -0.202 97.04%
q60 10.75 8.55 -2.205 79.50%
q61 10.08 7.88 -2.194 78.22%
q62 5.01 5.22 0.212 104.23%
q63 3.23 2.99 -0.236 92.70%
q64 64.96 64.08 -0.885 98.64%
q65 31.41 29.65 -1.754 94.42%
q66 4.98 4.93 -0.050 99.00%
q67 227.52 225.15 -2.376 98.96%
q68 5.16 4.29 -0.863 83.25%
q69 6.92 6.91 -0.007 99.90%
q70 12.79 12.74 -0.046 99.64%
q71 4.28 4.54 0.256 105.99%
q72 40.24 42.41 2.167 105.39%
q73 3.38 3.68 0.299 108.85%
q74 27.91 27.57 -0.340 98.78%
q75 43.10 44.85 1.748 104.06%
q76 15.04 14.10 -0.939 93.76%
q77 3.91 3.55 -0.355 90.91%
q78 85.34 84.79 -0.554 99.35%
q79 4.87 4.78 -0.094 98.08%
q80 17.08 17.25 0.172 101.01%
q81 8.91 9.87 0.955 110.72%
q82 10.48 10.33 -0.152 98.55%
q83 2.72 3.19 0.478 117.62%
q84 3.84 3.79 -0.040 98.94%
q85 9.66 10.26 0.599 106.20%
q86 4.62 4.52 -0.097 97.90%
q87 18.69 18.34 -0.342 98.17%
q88 23.39 24.05 0.662 102.83%
q89 4.28 4.86 0.583 113.62%
q90 3.83 3.37 -0.461 87.98%
q91 5.68 6.41 0.725 112.76%
q92 2.63 2.06 -0.565 78.50%
q93 54.69 55.27 0.581 101.06%
q94 18.06 17.27 -0.785 95.66%
q9 95.83 98.97 3.138 103.27%
q5 3.02 3.13 0.112 103.70%
q96 29.58 28.46 -1.118 96.22%
q97 3.40 2.77 -0.630 81.47%
q98 10.54 10.45 -0.082 99.22%
q99 10.54 10.45 -0.082 99.22%
total 2229.11 2197.23 -31.879 98.57%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_02_07_2025_time.csv log/native_master_02_06_2025_9aa21250fc_time.csv difference percentage
q1 42.37 43.76 1.396 103.29%
q2 43.16 45.60 2.441 105.66%
q3 93.39 92.21 -1.184 98.73%
q4 69.99 71.59 1.596 102.28%
q5 180.58 181.10 0.518 100.29%
q6 19.22 20.05 0.830 104.32%
q7 105.61 106.27 0.664 100.63%
q8 186.19 185.66 -0.523 99.72%
q9 282.10 280.98 -1.116 99.60%
q10 103.10 100.90 -2.198 97.87%
q11 34.18 33.85 -0.326 99.05%
q12 42.68 43.24 0.561 101.31%
q13 77.43 75.97 -1.458 98.12%
q14 35.55 37.72 2.170 106.10%
q15 67.19 66.87 -0.314 99.53%
q16 28.37 28.07 -0.300 98.94%
q17 232.96 233.46 0.498 100.21%
q18 338.82 359.75 20.929 106.18%
q19 36.32 38.15 1.829 105.04%
q20 62.00 64.46 2.465 103.98%
q21 530.75 535.62 4.864 100.92%
q22 24.84 24.71 -0.130 99.48%
total 2636.78 2669.99 33.212 101.26%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core VELOX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants