[SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys #47064

szehon-ho · 2024-06-22T01:52:24Z

What changes were proposed in this pull request?

This is the final planned SPJ scenario: auto-shuffle one side + less join keys than partition keys. Background:

Auto-shuffle works by creating ShuffleExchange for the non-partitioned side, with a clone of the partitioned side's KeyGroupedPartitioning.
"Less join key than partition key" works by 'projecting' all partition values by join keys (ie, keeping only partition columns that are join columns). It makes a target KeyGroupedShuffleSpec with 'projected' partition values, and then pushes this down to BatchScanExec. The BatchScanExec then 'groups' its projected partition value (except in the skew case but that's a different story..).

This combination is hard because the SPJ planning calls is spread in several places in this scenario. Given two sides, a non-partitioned side and a partitioned side, and the join keys are only a subset:

EnsureRequirements creates the target KeyGroupedShuffleSpec from the join's required distribution (ie, using only the join keys, not all partition keys).
EnsureRequirements copies this to the non-partitoned side's KeyGroupedPartition (for the auto-shuffle case)
BatchScanExec groups the partitions (for the partitioned side), including by join keys (if they differ from partition keys).

Take the example partition columns (id, name) , and partition values: (1, "bob"), (2, "alice"), (2, "sam").
Projection leaves us (1, 2, 2), and the final grouped partition values are (1, 2).

The problem is, that the two sides of the join do not match at all times. After the steps 1 and 2, the partitioned side has the 'projected' partition values (1, 2, 2), and the non-partitioned side creates a matching KeyGroupedPartitioning (1, 2, 2) for ShuffleExechange. But on step 3, the BatchScanExec for partitioned side groups the partitions to become (1, 2), but the non-partitioned side does not group and still retains (1, 2, 2) partitions. This leads to following assert error from the join:

requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.
java.lang.IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.
	at scala.Predef$.require(Predef.scala:337)
	at org.apache.spark.sql.catalyst.plans.physical.PartitioningCollection.<init>(partitioning.scala:550)
	at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning(ShuffledJoin.scala:49)
	at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning$(ShuffledJoin.scala:47)
	at org.apache.spark.sql.execution.joins.SortMergeJoinExec.outputPartitioning(SortMergeJoinExec.scala:39)
	at org.apache.spark.sql.execution.exchange.EnsureRequirements.$anonfun$ensureDistributionAndOrdering$1(EnsureRequirements.scala:66)
	at scala.collection.immutable.Vector1.map(Vector.scala:2140)
	at scala.collection.immutable.Vector1.map(Vector.scala:385)
	at org.apache.spark.sql.execution.exchange.EnsureRequirements.org$apache$spark$sql$execution$exchange$EnsureRequirements$$ensureDistributionAndOrdering(EnsureRequirements.scala:65)
	at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:657)
	at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:632)

The fix is to do the de-duplication in first pass.

Pushing down join keys to the BatchScanExec to return a de-duped outputPartitioning (partitioned side)
Creating the non-partitioned side's KeyGroupedPartitioning with de-duped partition keys (non-partitioned side).

Why are the changes needed?

This is the last planned scenario for SPJ not yet supported.

How was this patch tested?

Update existing unit test in KeyGroupedPartitionSuite

Was this patch authored or co-authored using generative AI tooling?

No.

szehon-ho

Some explanations

szehon-ho · 2024-06-22T01:56:57Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala

@@ -434,8 +434,13 @@ object KeyGroupedPartitioning {
    val projectedOriginalPartitionValues =
      originalPartitionValues.map(project(expressions, projectionPositions, _))

-    KeyGroupedPartitioning(projectedExpressions, projectedPartitionValues.length,
-      projectedPartitionValues, projectedOriginalPartitionValues)
+    val finalPartitionValues = projectedPartitionValues


This is de-duping the non-partitioned side (while creating the KeyGroupedPartition based on the partitioned side)

szehon-ho · 2024-06-22T01:57:21Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala

@@ -130,6 +130,15 @@ case class BatchScanExec(
        }
        k.copy(expressions = expressions, numPartitions = newPartValues.length,
          partitionValues = newPartValues)
+      case k: KeyGroupedPartitioning if spjParams.joinKeyPositions.isDefined =>


This is de-dupiing the outputPartitioning of the partitioned side.

I wonder if it is better to combine this clause with the clause of

case k: KeyGroupedPartitioning if spjParams.commonPartitionValues.isDefined

above, since there are some common logic. It's also worthwhile to add some comments here too, indicating that this case only applies when only one side of the join is KeyGroupedPartitioning and thus commonPartitionValues is not defined?

Combined in latest pr

szehon-ho · 2024-06-22T01:58:52Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala

@@ -279,7 +288,8 @@ case class StoragePartitionJoinParams(
    case other: StoragePartitionJoinParams =>
      this.commonPartitionValues == other.commonPartitionValues &&
      this.replicatePartitions == other.replicatePartitions &&
-      this.applyPartialClustering == other.applyPartialClustering
+      this.applyPartialClustering == other.applyPartialClustering &&
+      this.joinKeyPositions == other.joinKeyPositions


We've always wondered why these fields are not in equals. It turns out they have to be in order to properly run node.mapChildren() , otherwise fastEquals will determine the node is the same and skip, if these fields are different. Ref: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L1224

szehon-ho · 2024-06-22T02:04:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala

@@ -175,7 +175,11 @@ case class EnsureRequirements(
          child
        case ((child, dist), idx) =>
          if (bestSpecOpt.isDefined && bestSpecOpt.get.isCompatibleWith(specs(idx))) {
-            child
+            bestSpecOpt match {


This is pushing down the join key to the BatchScanExec of the partitioned side (to do de-duping).

Can we add some comments here? to explain this is the case when isKeyGroupCompatible is false, but we still want to avoid shuffling on one side?

Added comment

szehon-ho · 2024-06-24T16:25:48Z

@sunchao can you take a look?

sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala

sunchao · 2024-07-03T21:53:33Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala

@@ -130,6 +130,15 @@ case class BatchScanExec(
        }
        k.copy(expressions = expressions, numPartitions = newPartValues.length,
          partitionValues = newPartValues)
+      case k: KeyGroupedPartitioning if spjParams.joinKeyPositions.isDefined =>


I wonder if it is better to combine this clause with the clause of

case k: KeyGroupedPartitioning if spjParams.commonPartitionValues.isDefined

above, since there are some common logic. It's also worthwhile to add some comments here too, indicating that this case only applies when only one side of the join is KeyGroupedPartitioning and thus commonPartitionValues is not defined?

sunchao · 2024-07-03T22:00:34Z

sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala

@@ -175,7 +175,11 @@ case class EnsureRequirements(
          child
        case ((child, dist), idx) =>
          if (bestSpecOpt.isDefined && bestSpecOpt.get.isCompatibleWith(specs(idx))) {
-            child
+            bestSpecOpt match {


Can we add some comments here? to explain this is the case when isKeyGroupCompatible is false, but we still want to avoid shuffling on one side?

sunchao · 2024-07-03T22:02:20Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala

@@ -130,6 +130,15 @@ case class BatchScanExec(
        }
        k.copy(expressions = expressions, numPartitions = newPartValues.length,
          partitionValues = newPartValues)
+      case k: KeyGroupedPartitioning if spjParams.joinKeyPositions.isDefined =>
+        val expressions = spjParams.joinKeyPositions.get.map(i => k.expressions(i))
+        val newPartValues = k.partitionValues.map{r =>


Just curious why this is necessary. I see that in BatchScanExec we are also doing something similar, by checking the joinKeyPositions and call uniquePartitionValues at the end to de-dup.

Yea, it is the same code, let me know if you think of a way to code reuse.

The current uniquePartitionValues is called only at the end (the deferred grouping, while calculating the BatchScanExec's inputRDD), so its a bit too late. This same logic is needed in BatchScanExec.outputPartitioning (if its not the partiallyClusteredDistribution case), to make the two sides match during this scenario, after the first pass.

sunchao

LGTM, thanks @szehon-ho !

sunchao · 2024-07-12T21:21:32Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala

-  }
-
-
+      partitioning.expressions.forall{e =>


nit: space after forall

sunchao

LGTM, thanks @szehon-ho !

…s than partition keys ### What changes were proposed in this pull request? SPJ: Support auto-shuffle one side + less join keys than partition keys ### Why are the changes needed? This is the last planned scenario for SPJ not yet supported. ### How was this patch tested? Update existing unit test in KeyGroupedPartitionSuite ### Was this patch authored or co-authored using generative AI tooling? No.

szehon-ho · 2024-07-13T17:45:58Z

rebase

sunchao · 2024-07-14T23:53:59Z

Merge to master. Thanks!

szehon-ho · 2024-07-15T17:26:11Z

Thanks @sunchao !

…s than partition keys ### What changes were proposed in this pull request? This is the final planned SPJ scenario: auto-shuffle one side + less join keys than partition keys. Background: - Auto-shuffle works by creating ShuffleExchange for the non-partitioned side, with a clone of the partitioned side's KeyGroupedPartitioning. - "Less join key than partition key" works by 'projecting' all partition values by join keys (ie, keeping only partition columns that are join columns). It makes a target KeyGroupedShuffleSpec with 'projected' partition values, and then pushes this down to BatchScanExec. The BatchScanExec then 'groups' its projected partition value (except in the skew case but that's a different story..). This combination is hard because the SPJ planning calls is spread in several places in this scenario. Given two sides, a non-partitioned side and a partitioned side, and the join keys are only a subset: 1. EnsureRequirements creates the target KeyGroupedShuffleSpec from the join's required distribution (ie, using only the join keys, not all partition keys). 2. EnsureRequirements copies this to the non-partitoned side's KeyGroupedPartition (for the auto-shuffle case) 3. BatchScanExec groups the partitions (for the partitioned side), including by join keys (if they differ from partition keys). Take the example partition columns (id, name) , and partition values: (1, "bob"), (2, "alice"), (2, "sam"). Projection leaves us (1, 2, 2), and the final grouped partition values are (1, 2). The problem is, that the two sides of the join do not match at all times. After the steps 1 and 2, the partitioned side has the 'projected' partition values (1, 2, 2), and the non-partitioned side creates a matching KeyGroupedPartitioning (1, 2, 2) for ShuffleExechange. But on step 3, the BatchScanExec for partitioned side groups the partitions to become (1, 2), but the non-partitioned side does not group and still retains (1, 2, 2) partitions. This leads to following assert error from the join: ``` requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. java.lang.IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. at scala.Predef$.require(Predef.scala:337) at org.apache.spark.sql.catalyst.plans.physical.PartitioningCollection.<init>(partitioning.scala:550) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning(ShuffledJoin.scala:49) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning$(ShuffledJoin.scala:47) at org.apache.spark.sql.execution.joins.SortMergeJoinExec.outputPartitioning(SortMergeJoinExec.scala:39) at org.apache.spark.sql.execution.exchange.EnsureRequirements.$anonfun$ensureDistributionAndOrdering$1(EnsureRequirements.scala:66) at scala.collection.immutable.Vector1.map(Vector.scala:2140) at scala.collection.immutable.Vector1.map(Vector.scala:385) at org.apache.spark.sql.execution.exchange.EnsureRequirements.org$apache$spark$sql$execution$exchange$EnsureRequirements$$ensureDistributionAndOrdering(EnsureRequirements.scala:65) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:657) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:632) ``` The fix is to do the de-duplication in first pass. 1. Pushing down join keys to the BatchScanExec to return a de-duped outputPartitioning (partitioned side) 2. Creating the non-partitioned side's KeyGroupedPartitioning with de-duped partition keys (non-partitioned side). ### Why are the changes needed? This is the last planned scenario for SPJ not yet supported. ### How was this patch tested? Update existing unit test in KeyGroupedPartitionSuite ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47064 from szehon-ho/spj_less_join_key_auto_shuffle. Authored-by: Szehon Ho <[email protected]> Signed-off-by: Chao Sun <[email protected]>

…s than partition keys (apache#2005) ### What changes were proposed in this pull request? This is the final planned SPJ scenario: auto-shuffle one side + less join keys than partition keys. Background: - Auto-shuffle works by creating ShuffleExchange for the non-partitioned side, with a clone of the partitioned side's KeyGroupedPartitioning. - "Less join key than partition key" works by 'projecting' all partition values by join keys (ie, keeping only partition columns that are join columns). It makes a target KeyGroupedShuffleSpec with 'projected' partition values, and then pushes this down to BatchScanExec. The BatchScanExec then 'groups' its projected partition value (except in the skew case but that's a different story..). This combination is hard because the SPJ planning calls is spread in several places in this scenario. Given two sides, a non-partitioned side and a partitioned side, and the join keys are only a subset: 1. EnsureRequirements creates the target KeyGroupedShuffleSpec from the join's required distribution (ie, using only the join keys, not all partition keys). 2. EnsureRequirements copies this to the non-partitoned side's KeyGroupedPartition (for the auto-shuffle case) 3. BatchScanExec groups the partitions (for the partitioned side), including by join keys (if they differ from partition keys). Take the example partition columns (id, name) , and partition values: (1, "bob"), (2, "alice"), (2, "sam"). Projection leaves us (1, 2, 2), and the final grouped partition values are (1, 2). The problem is, that the two sides of the join do not match at all times. After the steps 1 and 2, the partitioned side has the 'projected' partition values (1, 2, 2), and the non-partitioned side creates a matching KeyGroupedPartitioning (1, 2, 2) for ShuffleExechange. But on step 3, the BatchScanExec for partitioned side groups the partitions to become (1, 2), but the non-partitioned side does not group and still retains (1, 2, 2) partitions. This leads to following assert error from the join: ``` requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. java.lang.IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. at scala.Predef$.require(Predef.scala:337) at org.apache.spark.sql.catalyst.plans.physical.PartitioningCollection.<init>(partitioning.scala:550) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning(ShuffledJoin.scala:49) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning$(ShuffledJoin.scala:47) at org.apache.spark.sql.execution.joins.SortMergeJoinExec.outputPartitioning(SortMergeJoinExec.scala:39) at org.apache.spark.sql.execution.exchange.EnsureRequirements.$anonfun$ensureDistributionAndOrdering$1(EnsureRequirements.scala:66) at scala.collection.immutable.Vector1.map(Vector.scala:2140) at scala.collection.immutable.Vector1.map(Vector.scala:385) at org.apache.spark.sql.execution.exchange.EnsureRequirements.org$apache$spark$sql$execution$exchange$EnsureRequirements$$ensureDistributionAndOrdering(EnsureRequirements.scala:65) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:657) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:632) ``` The fix is to do the de-duplication in first pass. 1. Pushing down join keys to the BatchScanExec to return a de-duped outputPartitioning (partitioned side) 2. Creating the non-partitioned side's KeyGroupedPartitioning with de-duped partition keys (non-partitioned side). ### Why are the changes needed? This is the last planned scenario for SPJ not yet supported. ### How was this patch tested? Update existing unit test in KeyGroupedPartitionSuite ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47064 from szehon-ho/spj_less_join_key_auto_shuffle. Authored-by: Szehon Ho <[email protected]> Signed-off-by: Chao Sun <[email protected]> Co-authored-by: Szehon Ho <[email protected]>

…s than partition keys ### What changes were proposed in this pull request? This is the final planned SPJ scenario: auto-shuffle one side + less join keys than partition keys. Background: - Auto-shuffle works by creating ShuffleExchange for the non-partitioned side, with a clone of the partitioned side's KeyGroupedPartitioning. - "Less join key than partition key" works by 'projecting' all partition values by join keys (ie, keeping only partition columns that are join columns). It makes a target KeyGroupedShuffleSpec with 'projected' partition values, and then pushes this down to BatchScanExec. The BatchScanExec then 'groups' its projected partition value (except in the skew case but that's a different story..). This combination is hard because the SPJ planning calls is spread in several places in this scenario. Given two sides, a non-partitioned side and a partitioned side, and the join keys are only a subset: 1. EnsureRequirements creates the target KeyGroupedShuffleSpec from the join's required distribution (ie, using only the join keys, not all partition keys). 2. EnsureRequirements copies this to the non-partitoned side's KeyGroupedPartition (for the auto-shuffle case) 3. BatchScanExec groups the partitions (for the partitioned side), including by join keys (if they differ from partition keys). Take the example partition columns (id, name) , and partition values: (1, "bob"), (2, "alice"), (2, "sam"). Projection leaves us (1, 2, 2), and the final grouped partition values are (1, 2). The problem is, that the two sides of the join do not match at all times. After the steps 1 and 2, the partitioned side has the 'projected' partition values (1, 2, 2), and the non-partitioned side creates a matching KeyGroupedPartitioning (1, 2, 2) for ShuffleExechange. But on step 3, the BatchScanExec for partitioned side groups the partitions to become (1, 2), but the non-partitioned side does not group and still retains (1, 2, 2) partitions. This leads to following assert error from the join: ``` requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. java.lang.IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. at scala.Predef$.require(Predef.scala:337) at org.apache.spark.sql.catalyst.plans.physical.PartitioningCollection.<init>(partitioning.scala:550) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning(ShuffledJoin.scala:49) at org.apache.spark.sql.execution.joins.ShuffledJoin.outputPartitioning$(ShuffledJoin.scala:47) at org.apache.spark.sql.execution.joins.SortMergeJoinExec.outputPartitioning(SortMergeJoinExec.scala:39) at org.apache.spark.sql.execution.exchange.EnsureRequirements.$anonfun$ensureDistributionAndOrdering$1(EnsureRequirements.scala:66) at scala.collection.immutable.Vector1.map(Vector.scala:2140) at scala.collection.immutable.Vector1.map(Vector.scala:385) at org.apache.spark.sql.execution.exchange.EnsureRequirements.org$apache$spark$sql$execution$exchange$EnsureRequirements$$ensureDistributionAndOrdering(EnsureRequirements.scala:65) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:657) at org.apache.spark.sql.execution.exchange.EnsureRequirements$$anonfun$1.applyOrElse(EnsureRequirements.scala:632) ``` The fix is to do the de-duplication in first pass. 1. Pushing down join keys to the BatchScanExec to return a de-duped outputPartitioning (partitioned side) 2. Creating the non-partitioned side's KeyGroupedPartitioning with de-duped partition keys (non-partitioned side). ### Why are the changes needed? This is the last planned scenario for SPJ not yet supported. ### How was this patch tested? Update existing unit test in KeyGroupedPartitionSuite ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47064 from szehon-ho/spj_less_join_key_auto_shuffle. Authored-by: Szehon Ho <[email protected]> Signed-off-by: Chao Sun <[email protected]>

github-actions bot added the SQL label Jun 22, 2024

szehon-ho commented Jun 22, 2024

View reviewed changes

szehon-ho force-pushed the spj_less_join_key_auto_shuffle branch from 8838f8d to 8940f6e Compare June 22, 2024 02:03

szehon-ho commented Jun 22, 2024

View reviewed changes

szehon-ho mentioned this pull request Jul 2, 2024

Spark 3.3: Support storage-partitioned joins apache/iceberg#6371

Merged

sunchao reviewed Jul 3, 2024

View reviewed changes

sunchao approved these changes Jul 12, 2024

View reviewed changes

szehon-ho added 3 commits July 13, 2024 10:45

Review comments

061bb3e

Add space

26589ec

szehon-ho force-pushed the spj_less_join_key_auto_shuffle branch from 190a76f to 26589ec Compare July 13, 2024 17:45

sunchao closed this in 206cc1a Jul 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys #47064

[SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys #47064

szehon-ho commented Jun 22, 2024 •

edited

Loading

szehon-ho left a comment

szehon-ho Jun 22, 2024

szehon-ho Jun 22, 2024

sunchao Jul 3, 2024

szehon-ho Jul 9, 2024

szehon-ho Jun 22, 2024

szehon-ho Jun 22, 2024

sunchao Jul 3, 2024

szehon-ho Jul 9, 2024

szehon-ho commented Jun 24, 2024

sunchao Jul 3, 2024

sunchao Jul 3, 2024

sunchao Jul 3, 2024

szehon-ho Jul 9, 2024 •

edited

Loading

sunchao left a comment

sunchao Jul 12, 2024

sunchao left a comment

szehon-ho commented Jul 13, 2024

sunchao commented Jul 14, 2024

szehon-ho commented Jul 15, 2024

[SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys #47064

[SPARK-48613][SQL] SPJ: Support auto-shuffle one side + less join keys than partition keys #47064

Conversation

szehon-ho commented Jun 22, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

szehon-ho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szehon-ho commented Jun 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szehon-ho Jul 9, 2024 • edited Loading

Choose a reason for hiding this comment

sunchao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunchao left a comment

Choose a reason for hiding this comment

szehon-ho commented Jul 13, 2024

sunchao commented Jul 14, 2024

szehon-ho commented Jul 15, 2024

szehon-ho commented Jun 22, 2024 •

edited

Loading

szehon-ho Jul 9, 2024 •

edited

Loading