Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-8581][VL] Fix Spark legacy date formatter under case insensitive configuration #8583

Merged
merged 4 commits into from
Jan 25, 2025

Conversation

weixiuli
Copy link
Contributor

What changes were proposed in this pull request?

The #7375 has enabled Velox simple date time formatter if the spark.sql.legacy.timeParserPolicy is set to LEGACY. But, it doesn't work when the spark.sql.legacy.timeParserPolicy is set to legacy.

(Fixes: #8581)

How was this patch tested?

Add unittests.

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Jan 21, 2025
Copy link

#8581

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

…rk.sql.legacy.timeParserPolicy is set to LEGACY or legacy.
Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

1 similar comment
Copy link

Run Gluten Clickhouse CI on x86

@PHILO-HE PHILO-HE changed the title [GLUTEN-8581][VL] Support Spark legacy date formatter whether the spark.sql.legacy.timeParserPolicy is set to LEGACY or legacy. [GLUTEN-8581][VL] Fix Spark legacy date formatter under case insensitive configuration Jan 22, 2025
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments. Thank you!

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

Copy link

Run Gluten Clickhouse CI on x86

@weixiuli
Copy link
Contributor Author

Copy link
Contributor

@zml1206 zml1206 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you.

Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@PHILO-HE PHILO-HE merged commit 70c3259 into apache:main Jan 25, 2025
50 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_01_25_2025_time.csv log/native_master_01_24_2025_beadc1a777_time.csv difference percentage
q1 16.11 15.58 -0.533 96.69%
q2 15.59 15.11 -0.475 96.95%
q3 3.56 3.68 0.125 103.51%
q4 85.74 85.32 -0.421 99.51%
q5 11.43 12.79 1.352 111.83%
q6 5.32 4.96 -0.351 93.39%
q7 7.94 8.64 0.698 108.79%
q8 4.94 4.85 -0.096 98.05%
q9 30.77 30.10 -0.671 97.82%
q10 12.29 11.92 -0.366 97.02%
q11 42.83 42.32 -0.506 98.82%
q12 2.18 2.05 -0.131 94.01%
q13 10.69 10.61 -0.088 99.18%
q14a 70.85 63.47 -7.379 89.58%
q14b 57.19 56.62 -0.566 99.01%
q15 3.82 4.04 0.222 105.82%
q16 29.25 28.93 -0.316 98.92%
q17 8.19 7.53 -0.655 92.00%
q18 10.02 10.77 0.757 107.56%
q19 3.74 5.25 1.506 140.22%
q20 2.75 2.01 -0.745 72.95%
q21 1.64 1.88 0.236 114.33%
q22 9.73 10.09 0.366 103.76%
q23a 139.84 139.86 0.023 100.02%
q23b 166.51 162.78 -3.731 97.76%
q24a 109.14 110.17 1.023 100.94%
q24b 97.46 96.69 -0.770 99.21%
q25 7.83 8.70 0.863 111.01%
q26 4.04 4.70 0.665 116.48%
q27 5.79 5.57 -0.216 96.26%
q28 38.07 36.80 -1.269 96.67%
q29 18.93 18.30 -0.630 96.67%
q30 6.45 6.58 0.122 101.89%
q31 11.92 12.03 0.110 100.92%
q32 2.29 2.36 0.071 103.11%
q33 6.96 7.45 0.488 107.01%
q34 4.45 4.48 0.036 100.81%
q35 11.21 9.97 -1.240 88.94%
q36 5.93 5.60 -0.327 94.49%
q37 5.27 5.37 0.097 101.83%
q38 17.45 17.41 -0.041 99.77%
q39a 4.71 4.31 -0.398 91.54%
q39b 4.94 5.05 0.117 102.37%
q40 5.24 5.41 0.171 103.27%
q41 0.89 1.19 0.306 134.47%
q42 1.14 1.31 0.172 115.10%
q43 4.45 4.52 0.068 101.52%
q44 11.60 11.49 -0.110 99.05%
q45 4.80 4.12 -0.682 85.78%
q46 5.59 5.42 -0.169 96.98%
q47 24.12 19.56 -4.560 81.10%
q48 7.13 6.65 -0.488 93.16%
q49 9.96 10.67 0.710 107.13%
q50 38.76 38.54 -0.217 99.44%
q51 14.55 14.50 -0.047 99.68%
q52 1.27 1.18 -0.085 93.29%
q53 3.15 2.98 -0.169 94.62%
q54 6.74 6.77 0.030 100.45%
q55 1.47 1.48 0.008 100.56%
q56 7.38 6.98 -0.399 94.60%
q57 13.16 13.31 0.145 101.10%
q58 3.40 3.46 0.055 101.63%
q59 6.66 7.27 0.611 109.18%
q60 8.38 10.74 2.362 128.18%
q61 8.56 9.60 1.037 112.12%
q62 5.31 4.92 -0.395 92.56%
q63 3.53 3.07 -0.463 86.90%
q64 66.35 60.51 -5.841 91.20%
q65 29.79 29.78 -0.013 99.96%
q66 4.60 4.53 -0.075 98.37%
q67 227.98 226.87 -1.112 99.51%
q68 4.60 4.09 -0.516 88.80%
q69 6.94 7.01 0.071 101.03%
q70 12.50 14.72 2.218 117.74%
q71 6.20 4.26 -1.934 68.79%
q72 39.70 38.09 -1.612 95.94%
q73 3.40 3.19 -0.210 93.83%
q74 28.15 26.97 -1.182 95.80%
q75 42.73 43.84 1.107 102.59%
q76 15.07 14.35 -0.714 95.26%
q77 3.61 3.29 -0.311 91.38%
q78 84.06 84.25 0.189 100.23%
q79 5.23 5.22 -0.002 99.96%
q80 17.28 17.40 0.127 100.74%
q81 8.88 8.02 -0.863 90.28%
q82 10.57 10.01 -0.566 94.64%
q83 2.91 2.72 -0.196 93.28%
q84 4.00 4.23 0.234 105.85%
q85 9.64 10.15 0.516 105.36%
q86 4.57 4.48 -0.096 97.89%
q87 18.84 17.83 -1.019 94.59%
q88 23.47 24.08 0.608 102.59%
q89 4.76 4.64 -0.122 97.44%
q90 3.29 3.47 0.181 105.50%
q91 5.34 6.44 1.097 120.54%
q92 2.27 2.51 0.234 110.31%
q93 54.95 52.07 -2.888 94.74%
q94 17.00 17.68 0.679 103.99%
q9 96.44 101.04 4.603 104.77%
q5 3.84 3.11 -0.725 81.09%
q96 27.88 28.63 0.744 102.67%
q97 2.67 2.61 -0.051 98.08%
q98 10.02 10.48 0.461 104.60%
q99 10.02 10.48 0.461 104.60%
total 2222.54 2200.41 -22.129 99.00%

baibaichen pushed a commit to baibaichen/gluten that referenced this pull request Feb 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core VELOX
Projects
None yet
7 participants