[SPARK-23500][SQL][FOLLOWUP] Fix complex type simplification rules to apply to entire plan #20911
Conversation
| import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan | ||
| import org.apache.spark.sql.test.SharedSQLContext | ||
|
|
||
| class ComplexTypesSuite extends QueryTest with SharedSQLContext { |
There was a problem hiding this comment.
Currently, it majorly covers the end-to-end test cases of named_struct. In the future, we can add more about the others.
|
Test build #88621 has finished for PR 20911 at commit
|
| } | ||
|
|
||
| test("SPARK-23500: do not simplify maps in Aggregate expressions") { | ||
| // This could be done if we had a more complex rule that checks that |
There was a problem hiding this comment.
shall we merge it with the previous test case?
| numFields = 1, | ||
| containsNull = false), | ||
| ordinal = 1) as "a2") | ||
| .where('id > 0L) |
There was a problem hiding this comment.
shall we use sort? We are trying to test when the complex expressions are not in the root node. It's hard to say if we push filter through project or not.
There was a problem hiding this comment.
sure, let me get rid of it.
|
Test build #88654 has finished for PR 20911 at commit
|
|
retest this please |
|
Test build #88658 has finished for PR 20911 at commit
|
|
Test build #88720 has finished for PR 20911 at commit
|
| .select( | ||
| GetMapValue('m, "r1") as "a1", | ||
| GetMapValue('m, "r32") as "a2") | ||
| .where('id > 0L) |
There was a problem hiding this comment.
can we use sort here too?
|
LGTM |
|
Test build #88740 has finished for PR 20911 at commit
|
|
thanks, merging to master! |
… apply to entire plan ## What changes were proposed in this pull request? This PR is to improve the test coverage of the original PR apache#20687 ## How was this patch tested? N/A Author: gatorsmile <[email protected]> Closes apache#20911 from gatorsmile/addTests.
What changes were proposed in this pull request?
This PR is to improve the test coverage of the original PR #20687
How was this patch tested?
N/A