[VL] Iceberg bucket scan failed

[FelixYBW]

Backend

VL (Velox)

Bug description

"spark.sql.sources.v2.bucketing.pushPartValues.enabled": "false",
"spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled": "false"


Py4JJavaError: An error occurred while calling o401.count.
: org.apache.gluten.exception.GlutenException: The partition length of all the leaf transformer are not the same.
	at org.apache.gluten.execution.WholeStageTransformer.getSplitInfosFromPartitions(WholeStageTransformer.scala:508)
	at org.apache.gluten.execution.WholeStageTransformer.doExecuteColumnar(WholeStageTransformer.scala:379)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:222)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
	at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:218)
	at org.apache.gluten.execution.VeloxResizeBatchesExec.doExecuteColumnar(VeloxResizeBatchesExec.scala:70)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeColumnar$1(SparkPlan.scala:222)
	at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:246)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:243)
	at org.apache.spark.sql.execution.SparkPlan.executeColumnar(SparkPlan.scala:218)

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

[jinchengchenghh]

I will fallback when spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled is true, and try to fix this issue.
I have reproduced by UT.