Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update correlation in Apply node #5687

Closed
wants to merge 7 commits into from

Conversation

kokosing
Copy link
Contributor

@kokosing kokosing commented Jul 18, 2016

Update correlation in Apply node

All the column references of subquery plan which are not resolved (do
not come from subquery relations) are collected. SubqueryPlanner check
if it is able to rewrite such references to symbol produced by its
relation, if it is able then add replace column reference with that
symbol and that this symbol to the correlation list of Apply node. If
SubqueryPlanner is not able to rewrite that column reference, then it is
assumed the outer SubqueryPlanner execution will do that. If there is no
outer SubqueryPlanner then SemanticException should be thrown.

@kokosing kokosing changed the title Add unit tests for correlated EXISTS subquery Update correlation in Apply node Jul 18, 2016
@kokosing
Copy link
Contributor Author

Supersedes #5302.
Please notice that this pull request is based on #5686

@kokosing kokosing force-pushed the feature_correlation branch from 0e099ff to 1747df3 Compare July 18, 2016 08:40
@ghost ghost added the CLA Signed label Jul 21, 2016
@ghost ghost added the CLA Signed label Aug 1, 2016
@ghost ghost added the CLA Signed label Aug 2, 2016
@ghost ghost added the CLA Signed label Aug 2, 2016
kokosing added a commit to Teradata/presto that referenced this pull request Aug 2, 2016
@ghost ghost added the CLA Signed label Aug 2, 2016

checkState(resolvedField.isPresent() && resolvedField.get().isLocal(), "Unable to get symbol for field in outer scope: '%s'", expression);
if (resolvedField.isPresent() && resolvedField.get().isLocal()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return resolvedField.filter(ResolvedField::isLocal).map(field -> outputSymbols.get(field.getFieldIndex()))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@haozhun
Copy link
Contributor

haozhun commented Aug 8, 2016

I did not read this pull request. But I would like to add a comment because of a production issue we've seen before. When Teradata brought non-equi conjunct support to outer join in Presto (thank you again), we removed the sanity check that fails any query that contains such conjuncts (and its tests) before the feature were linked together to execute such query correctly. As a result, for more than a month, the Presto release version produces incorrect result for such queries.

Please make sure any query with correlated subqueries continues to fail loudly until such query produces correct result.

@kokosing
Copy link
Contributor Author

kokosing commented Aug 9, 2016

@haozhun That is a general true. None of the features should be enabled until it works correctly. I am trying to test it as mush as I can, nevertheless the bugs may still exists. That is why code review is so much important, that I could benefit from your experience and possibly broaden the test coverage to avoid that situation you (FB) experienced before.

@ghost ghost added the CLA Signed label Aug 9, 2016
@kokosing
Copy link
Contributor Author

kokosing commented Aug 9, 2016

@haozhun Maybe we could display a notice to the user about possible instability of new features, that given feature was recently added and may still have bugs or instabilities. Such notice could be displayed through couple of Presto releases. What do you think about such approach, would it help if you had something like that for non-equi conjunct support to outer join back then?

@kokosing kokosing force-pushed the feature_correlation branch from 1747df3 to e0a34b7 Compare August 9, 2016 07:19
@@ -70,6 +70,11 @@

public class LogicalPlanner
{
public enum Stage
{
CREATED, OPTIMIZED, OPTIMIZED_AND_VALIDATED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably in a separate commit.

Also, can you remind me of the rationale for this? I remember discussing about this, but I can't find the original comments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. I added a commit message which explains the motivation.

@kokosing
Copy link
Contributor Author

@martint Rebased. Comments applied. Superseded by #5943

@ghost ghost added the CLA Signed label Aug 24, 2016
@kokosing kokosing closed this Aug 24, 2016
kokosing added a commit to Teradata/presto that referenced this pull request Aug 26, 2016
kokosing added a commit to Teradata/presto that referenced this pull request Aug 26, 2016
martint pushed a commit to martint/presto-facebook that referenced this pull request Aug 26, 2016
@kokosing kokosing deleted the feature_correlation branch August 29, 2016 05:54
losipiuk pushed a commit to losipiuk/prestodb that referenced this pull request Sep 8, 2016
losipiuk pushed a commit to losipiuk/prestodb that referenced this pull request Sep 12, 2016
losipiuk pushed a commit to Teradata/presto that referenced this pull request Sep 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants