Improve performance of dropDuplicates

[andygrove]

What is the problem the feature request solves?

Comet is 4x slower than Spark for the following query:

spark.read.parquet("/mnt/bigdata/tpcds/sf100/store_sales.parquet") \
    .repartition("ss_item_sk") \
    .dropDuplicates(["ss_item_sk", "ss_quantity"]) \
    .write.parquet("output.parquet")

If I remove the dropDuplicates call, I see similar performance between Comet and Spark.

Describe the potential solution

No response

Additional context

No response

[andygrove]

Disabling Comet aggregation improves performance from 230s to 64s and is slightly faster than Spark at 78s.