Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-17337][SQL] Do not pushdown predicates through filters with predicate subqueries #15761

Closed
wants to merge 2 commits into from

Conversation

hvanhovell
Copy link
Contributor

@hvanhovell hvanhovell commented Nov 3, 2016

What changes were proposed in this pull request?

The PushDownPredicate rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source).

The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan.

How was this patch tested?

Added a test to SubquerySuite. @nsyca has done previous work this. I have taken test from his initial PR.

@nsyca
Copy link
Contributor

nsyca commented Nov 4, 2016

A patch nicely done on this local issue. Minimal code but effective. The bigger problem remains there though, tracked by SPARk-17154.

@SparkQA
Copy link

SparkQA commented Nov 4, 2016

Test build #68094 has finished for PR 15761 at commit 6993c50.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@hvanhovell hvanhovell changed the title [SPARK-17337][SQL] Do not pushdown predicates through filters with predicate subqueries [WIP] [SPARK-17337][SQL] Do not pushdown predicates through filters with predicate subqueries Nov 4, 2016
@SparkQA
Copy link

SparkQA commented Nov 4, 2016

Test build #68145 has finished for PR 15761 at commit 25970c5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@sameeragarwal sameeragarwal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hvanhovell
Copy link
Contributor Author

hvanhovell commented Nov 4, 2016

Merging this to master/branch-2.1. Thanks for the reviews!

I will open a new PR for branch 2.0.

asfgit pushed a commit that referenced this pull request Nov 4, 2016
…edicate subqueries

## What changes were proposed in this pull request?
The `PushDownPredicate` rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source).

The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan.

## How was this patch tested?
Added a test to `SubquerySuite`. nsyca has done previous work this. I have taken test from his initial PR.

Author: Herman van Hovell <[email protected]>

Closes #15761 from hvanhovell/SPARK-17337.

(cherry picked from commit 550cd56)
Signed-off-by: Herman van Hovell <[email protected]>
@asfgit asfgit closed this in 550cd56 Nov 4, 2016
asfgit pushed a commit that referenced this pull request Nov 4, 2016
… PRs

## What changes were proposed in this pull request?
This PR backports two subquery related PRs to branch-2.0:

- #14411
- #15761

## How was this patch tested?
Added a tests  to `SubquerySuite`.

Author: Nattavut Sutyanyong <[email protected]>
Author: Herman van Hovell <[email protected]>

Closes #15772 from hvanhovell/SPARK-17337-2.0.
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
…edicate subqueries

## What changes were proposed in this pull request?
The `PushDownPredicate` rule can create a wrong result if we try to push a filter containing a predicate subquery through a project when the subquery and the project share attributes (have the same source).

The current PR fixes this by making sure that we do not push down when there is a predicate subquery that outputs the same attributes as the filters new child plan.

## How was this patch tested?
Added a test to `SubquerySuite`. nsyca has done previous work this. I have taken test from his initial PR.

Author: Herman van Hovell <[email protected]>

Closes apache#15761 from hvanhovell/SPARK-17337.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants