[CORE] Fix ut module compilation with hadoop3.x

[Yohahaha]

mvn clean package -P backends-velox,spark-3.3,spark-ut,hadoop-3.2 -DskipTests

[ERROR] /root/oap-gluten/gluten-ut/spark33/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetFilterSuite.scala:617: error: reference to In is ambiguous;
[ERROR] it is imported twice in the same scope by
[ERROR] import org.apache.parquet.filter2.predicate.Operators.{Column=>_, _}
[ERROR] and import org.apache.spark.sql.catalyst.expressions._
[ERROR]                           In(
[ERROR]                           ^
[ERROR] one error found
[ERROR] exception compilation error occurred!!!

Fix above compilation error and remove specified parquet-column dep, remove unnecessary arrow-version.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI

[ulysses-you]

it requires upgrade parquet version..

[Yohahaha]

it requires upgrade parquet version..

I see, CI use -Phadoop2.7, I think we need set default profile to hadoop3 at least to align with Spark. I will open a issue for this.

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

@Yohahaha LGTM. Would you like to add the Maven build command you used to produce this issue to PR description?

[Yohahaha]

@Yohahaha LGTM. Would you like to add the Maven build command you used to produce this issue to PR description?

added, thanks!

[zhztheplayer]

@Yohahaha LGTM. Would you like to add the Maven build command you used to produce this issue to PR description?

added, thanks!

Thanks. Why is the issue not failing CI?

[Yohahaha]

@Yohahaha LGTM. Would you like to add the Maven build command you used to produce this issue to PR description?

added, thanks!

Thanks. Why is the issue not failing CI?

Due to this #4491, Gluten's default hadoop version is 2.7.

I see, CI use -Phadoop2.7, I think we need set default profile to hadoop3 at least to align with Spark. I will open a issue for this.

[GlutenPerfBot]

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_4490_time.csv	log/native_master_01_23_2024_056512473_time.csv	difference	percentage
q1	33.49	33.97	0.486	101.45%
q2	23.89	23.91	0.018	100.07%
q3	38.53	36.73	-1.798	95.33%
q4	37.45	37.55	0.097	100.26%
q5	67.92	68.43	0.516	100.76%
q6	7.39	6.94	-0.453	93.87%
q7	82.51	81.76	-0.747	99.09%
q8	83.15	82.93	-0.217	99.74%
q9	123.76	116.13	-7.625	93.84%
q10	42.07	42.50	0.432	101.03%
q11	19.55	19.93	0.386	101.98%
q12	25.19	25.93	0.744	102.95%
q13	44.89	44.75	-0.132	99.71%
q14	15.17	19.50	4.331	128.56%
q15	28.76	30.82	2.059	107.16%
q16	14.19	13.93	-0.264	98.14%
q17	98.54	97.72	-0.821	99.17%
q18	146.83	148.73	1.893	101.29%
q19	15.28	12.44	-2.842	81.40%
q20	26.83	26.84	0.012	100.05%
q21	224.39	223.34	-1.057	99.53%
q22	13.64	13.33	-0.312	97.71%
total	1213.42	1208.13	-5.292	99.56%