fix: [comet-parquet-exec] fix regressions original comet native scal implementation #1170
Conversation
…TIVE_SCAN is enabled
andygrove
changed the title
| Preconditions.checkState( | ||
| t.isPrimitive() && !t.isRepetition(Type.Repetition.REPEATED), | ||
| "Complex type is not supported"); | ||
| // Preconditions.checkState( |
There was a problem hiding this comment.
This is for the second implementation. Will remove this later
| @@ -39,7 +39,10 @@ trait DataTypeSupport { | |||
| BinaryType | StringType | _: DecimalType | DateType | TimestampType => | |||
| true | |||
| case t: DataType if t.typeName == "timestamp_ntz" => true | |||
| case _: StructType => true | |||
| case _: StructType | |||
There was a problem hiding this comment.
With only the original scan enabled, this caused CometScan to be used for Struct types (which it did not support).
| @@ -131,8 +131,15 @@ case class CometScanExec( | |||
| // exposed for testing | |||
| lazy val bucketedScan: Boolean = wrapped.bucketedScan | |||
|
|
|||
| override lazy val (outputPartitioning, outputOrdering): (Partitioning, Seq[SortOrder]) = | |||
| (wrapped.outputPartitioning, wrapped.outputOrdering) | |||
| override lazy val (outputPartitioning, outputOrdering): (Partitioning, Seq[SortOrder]) = { | |||
There was a problem hiding this comment.
@viirya - The previous fix to address outputPartitioning (using inputRDD) was not correct and caused multiple test failures. This is a different attempt (and at least all the tests pass).
If this is a bucketedScan, we fall back to the wrapped FileSourceScanLike implementation but for non bucketed case since FileSourceScanLike always returned 0 partitions, we override the behaviour, setting the num of partitions to the number of files.
I'm not entirely sure this covers all cases so please advise.
Changes in CometScanExec to support the full native reader caused the original reader. This fixes the issues so that all Comet unit tests pass with the original native scan. The only tests that do not pass are the plan stability tests where the 'expected' plans expect
CometNativeScanbut if full native is not turned on, the plan produces a CometScan (which is expected).