[SPARK-5649][SQL] added a rule to check datatypes cast

[scwf]

Throw an exception when datatypes cast is illegitimate, this patch info user the cast issue in the sqls.
Before this patch, we only get an attribute unresolved exception which is really hard to understand the problems in users' sql

[SparkQA]

Test build #26908 has started for PR 4425 at commit 9fb60ce.

• This patch merges cleanly.

[SparkQA]

Test build #26908 has finished for PR 4425 at commit 9fb60ce.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26908/
Test FAILed.

[SparkQA]

Test build #26910 has started for PR 4425 at commit 172e4dd.

• This patch merges cleanly.

[SparkQA]

Test build #26911 has started for PR 4425 at commit 97bb90b.

• This patch merges cleanly.

[SparkQA]

Test build #26910 has finished for PR 4425 at commit 172e4dd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26910/
Test PASSed.

[SparkQA]

Test build #26911 has finished for PR 4425 at commit 97bb90b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26911/
Test PASSed.

[chenghao-intel]

It should be either a bug of the Analyzer or the bad semantic of the user. As I think all of the children should be resolved before the checking, can you also give an example or unit test that will throw exception?

[scwf]

unit test added

[SparkQA]

Test build #26917 has started for PR 4425 at commit bd8d9f8.

• This patch merges cleanly.

[SparkQA]

Test build #26917 has finished for PR 4425 at commit bd8d9f8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26917/
Test PASSed.

[scwf]

we can throw an AnalysisException not treenodeexception after #4439 has in

[SparkQA]

Test build #27260 has started for PR 4425 at commit db86dc4.

• This patch merges cleanly.

[SparkQA]

Test build #27260 has finished for PR 4425 at commit db86dc4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27260/
Test FAILed.

[scwf]

Retest this please

[SparkQA]

Test build #27266 has started for PR 4425 at commit db86dc4.

• This patch merges cleanly.

[SparkQA]

Test build #27266 has finished for PR 4425 at commit db86dc4.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27266/
Test FAILed.

[scwf]

retest this please

[SparkQA]

Test build #27272 has started for PR 4425 at commit db86dc4.

• This patch merges cleanly.

[SparkQA]

Test build #27272 has finished for PR 4425 at commit db86dc4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27272/
Test PASSed.

[scwf]

Updated, @marmbrus any comment here?

[marmbrus]

I think we want to check casts after we check resolution. Otherwise casts of invalid attributes will still give confusing TreeNodeExceptions.

scala> sql("SELECT CAST(x AS STRING) FROM src").collect()
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'x
    at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:48)

[scwf]

If we check resolution first,
1 check cast will be invalid since legitimate cast expression must be not resolved.
2 select cast(key as binary) from src will give a scala.NotImplementedError

how about not change the order but catch exception of child.dataType to make it pass this rule? then your case will go into check resolution

[scwf]

scala.NotImplementedError should be a bug, filed PR #4552 for this

[marmbrus]

Thanks for bringing this up! I rolled the fix into #4558.