-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-17337][SQL] Do not pushdown predicates through filters with predicate subqueries #15761
Conversation
A patch nicely done on this local issue. Minimal code but effective. The bigger problem remains there though, tracked by SPARk-17154. |
Test build #68094 has finished for PR 15761 at commit
|
Test build #68145 has finished for PR 15761 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Merging this to master/branch-2.1. Thanks for the reviews! I will open a new PR for branch 2.0. |
…edicate subqueries ## What changes were proposed in this pull request? The `PushDownPredicate` rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source). The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan. ## How was this patch tested? Added a test to `SubquerySuite`. nsyca has done previous work this. I have taken test from his initial PR. Author: Herman van Hovell <[email protected]> Closes #15761 from hvanhovell/SPARK-17337. (cherry picked from commit 550cd56) Signed-off-by: Herman van Hovell <[email protected]>
… PRs ## What changes were proposed in this pull request? This PR backports two subquery related PRs to branch-2.0: - #14411 - #15761 ## How was this patch tested? Added a tests to `SubquerySuite`. Author: Nattavut Sutyanyong <[email protected]> Author: Herman van Hovell <[email protected]> Closes #15772 from hvanhovell/SPARK-17337-2.0.
…edicate subqueries ## What changes were proposed in this pull request? The `PushDownPredicate` rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source). The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan. ## How was this patch tested? Added a test to `SubquerySuite`. nsyca has done previous work this. I have taken test from his initial PR. Author: Herman van Hovell <[email protected]> Closes apache#15761 from hvanhovell/SPARK-17337.
What changes were proposed in this pull request?
The
PushDownPredicate
rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source).The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan.
How was this patch tested?
Added a test to
SubquerySuite
. @nsyca has done previous work this. I have taken test from his initial PR.