Result mismatch when data contains empty map

[yma11]

Backend

VL (Velox)

Bug description

UT for reproduce:

    withTempPath {
      path =>
        Seq(
          Map[Int, String](1 -> null, 2 -> "200"),
          Map[Int, String](),
          Map[Int, String](1 -> "100", 2 -> "200", 3 -> "300"),
          null
        )
          .toDF("i")
          .write
          .parquet(path.getCanonicalPath)

        spark.read.parquet(path.getCanonicalPath).createOrReplaceTempView("map_tbl")

        runQueryAndCompare("select map_entries(i) from map_tbl") {
          checkOperatorMatch[ProjectExecTransformer]
        }
    }

Result mismatch:

== Physical Plan ==
*(1) Project [map_entries(i#144) AS map_entries(i)#152]
+- VeloxColumnarToRowExec
   +- ^(1) BatchScanExecTransformer[i#144] ParquetScan DataFilters: [], Format: parquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/spark-f49a09d8-ad14-445a-a536-ca9f802dd495], PartitionFilters: [], PushedAggregation: [], PushedFilters: [], PushedGroupBy: [], ReadSchema: struct<i:map<int,string>>, PushedFilters: [], PushedAggregation: [], PushedGroupBy: [] RuntimeFilters: []

== Results ==

== Results ==
!== Correct Answer - 4 ==                   == Gluten Answer - 4 ==
 struct<>                                                struct<>
 [ArrayBuffer()]                                       [ArrayBuffer()]
![ArrayBuffer([1,100], [2,200], [3,300])]   [ArrayBuffer([0,null], [1,100], [2,200])]
 [ArrayBuffer([1,null], [2,200])]                [ArrayBuffer([1,null], [2,200])]
 [null]                                                      [null]
    

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[Yohahaha]

facebookincubator/velox#7776

[zhouyuan]

related: #4778

[ulysses-you]

@yma11 @zhouyuan does this issue still exist ? it seems has been fixed by facebookincubator/velox#9187

[yma11]

Thanks for ping. Let me have a check.

[yma11]

Verified using the UT.

[ulysses-you]

thank you @yma11 , shall we tune spark.gluten.sql.complexType.scan.fallback.enabled to false by default ?

[yma11]

I think for customer, they can take a try to disable if there is no result mismatch issue in there scenario. We are still not so confident about complex type support in velox.