[SPARK-33400][SQL] Normalize sameOrderExpressions in SortOrder to avoid unnecessary sort operations

[prakharjain09]

What changes were proposed in this pull request?

This pull request tries to normalize the SortOrder properly to prevent unnecessary sort operators. Currently the sameOrderExpressions are not normalized as part of AliasAwareOutputOrdering.

Example: consider this join of three tables:

  """
    |SELECT t2id, t3.id as t3id
    |FROM (
    |    SELECT t1.id as t1id, t2.id as t2id
    |    FROM t1, t2
    |    WHERE t1.id = t2.id
    |) t12, t3
    |WHERE t1id = t3.id
  """.

The plan for this looks like:

  *(8) Project [t2id#1059L, id#1004L AS t3id#1060L]
  +- *(8) SortMergeJoin [t2id#1059L], [id#1004L], Inner
     :- *(5) Sort [t2id#1059L ASC NULLS FIRST ], false, 0         <-----------------------------
     :  +- *(5) Project [id#1000L AS t2id#1059L]
     :     +- *(5) SortMergeJoin [id#996L], [id#1000L], Inner
     :        :- *(2) Sort [id#996L ASC NULLS FIRST ], false, 0
     :        :  +- Exchange hashpartitioning(id#996L, 5), true, [id=#1426]
     :        :     +- *(1) Range (0, 10, step=1, splits=2)
     :        +- *(4) Sort [id#1000L ASC NULLS FIRST ], false, 0
     :           +- Exchange hashpartitioning(id#1000L, 5), true, [id=#1432]
     :              +- *(3) Range (0, 20, step=1, splits=2)
     +- *(7) Sort [id#1004L ASC NULLS FIRST ], false, 0
        +- Exchange hashpartitioning(id#1004L, 5), true, [id=#1443]
           +- *(6) Range (0, 30, step=1, splits=2)

In this plan, the marked sort node could have been avoided as the data is already sorted on "t2.id" by the lower SortMergeJoin.

Why are the changes needed?

To remove unneeded Sort operators.

Does this PR introduce any user-facing change?

No

How was this patch tested?

New UT added.

[maropu]

This is related to #30300 ?

[prakharjain09]

@maropu I have updated the pull request to cover one scenario which was not covered by #30300 . Please review the changes.

[prakharjain09]

cc - @cloud-fan @imback82

[maropu]

ok to test

[maropu]

LGTM if the existing tests pass.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35839/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35839/

[SparkQA]

Test build #131235 has finished for PR 30302 at commit 997f98c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SubExprEvaluationRuntime(cacheMaxEntries: Int)
• case class ExpressionProxy(
• case class ResultProxy(result: Any)

[prakharjain09]

@maropu @cloud-fan Thanks for reviewing the changes. Please merge the changes if there are no further comments.

I will work on a followup PR to make sameOrderExpressions child of SortOrder.

[cloud-fan]

thanks, merging to master!