[SQL][SPARK-7088] Fix analysis for 3rd party logical plan. #6853
Conversation
|
This is still a work in progress, since I have not tested yet. |
|
If I understand correctly, you are just delaying the failure until |
|
ok to test |
|
Test build #35063 has finished for PR 6853 at commit
|
|
@smola can you put WIP on the title until you feel it is ready? |
smola
changed the title
|
@marmbrus Yes, this patch is meant just to delay the check until check analysis. The reason is that just because ResolveReferences rule cannot resolve the plan, that does not mean that there is no other rule resolving it. I think this is the main idea behind how rules work in catalyst, right? Each rule takes care of what it knows and ignores the unknown. With respect my use case, my custom logical plan does produce new attributes. I have also added resolution rules for it on my side. So yes, analysis is checked. But the current problem with this is that I need to maintain a copy of ResolveReferences (i.e. FixedResolveReferences) in my code, instead of just adding my new logic in ResolveMyCustomPlan. Then I have to override SQLContext and the analyzer just to be able to replace the default rule with mine. |
| case j @ Join(left, right, _, _) if left.outputSet.intersect(right.outputSet).nonEmpty => | ||
| val conflictingAttributes = left.outputSet.intersect(right.outputSet) | ||
| failAnalysis(s""" | ||
| |Failure when resolving conflicting references in Join: |
|
Thanks for the explanation, that does sound reasonable to me. A few minor suggestions otherwise LGTM. |
ResolveReferences analysis rule now does not throw when it cannot resolve references in a self-join.
smola
changed the title
|
Test build #35698 has finished for PR 6853 at commit
|
|
Thanks! Merged to master. |
| @@ -121,6 +122,17 @@ trait CheckAnalysis { | |||
|
|
|||
| case _ => // Analysis successful! | |||
| } | |||
|
|
|||
| // Special handling for cases when self-join introduce duplicate expression ids. | |||
| case j @ Join(left, right, _, _) if left.outputSet.intersect(right.outputSet).nonEmpty => | |||
There was a problem hiding this comment.
this is not reachable according to the compiler. @smola can you submit a fix for this? thanks.
There was a problem hiding this comment.
NAVER - http://www.naver.com/
[email protected] 님께 보내신 메일 <Re: [spark] [SQL][SPARK-7088] Fix analysis for 3rd party logical plan. (#6853)> 이 다음과 같은 이유로 전송 실패했습니다.
받는 사람이 회원님의 메일을 수신차단 하였습니다.
ResolveReferences analysis rule now does not throw when it cannot resolve references in a self-join.