[SPARK-14349] [SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. #12134

gatorsmile · 2016-04-03T00:16:05Z

What changes were proposed in this pull request?

Currently, the weird error messages are issued if we use Hive Context-only operations in SQL Context.

For example,

When calling Drop Table in SQL Context, we got the following message:

Expected exception org.apache.spark.sql.catalyst.parser.ParseException to be thrown, but java.lang.ClassCastException was thrown.

When calling Script Transform in SQL Context, we got the message:

assertion failed: No plan for ScriptTransformation [key#9,value#10], cat, [tKey#155,tValue#156], null
+- LogicalRDD [key#9,value#10], MapPartitionsRDD[3] at beforeAll at BeforeAndAfterAll.scala:187

Updates:
Based on the investigation from @hvanhovell , the root cause is visitChildren, which is the default implementation. It always returns the result of the last defined context child. After merging the code changes from @hvanhovell , it works! Thank you @hvanhovell !

How was this patch tested?

A few test cases are added.

Not sure if the same issue exist for the other operators/DDL/DML. @hvanhovell

SparkQA · 2016-04-04T08:27:43Z

Test build #54835 has finished for PR 12134 at commit c7c9461.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-04T09:36:50Z

Test build #54838 has finished for PR 12134 at commit e69a7e8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-04-04T13:35:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

      inRowFormat: RowFormatContext,
      recordWriter: Token,
      outRowFormat: RowFormatContext,
      recordReader: Token,
-      schemaLess: Boolean): ScriptInputOutputSchema = null


Undo this change and throw the exception here.

Done. Thank you!

SparkQA · 2016-04-04T16:01:17Z

Test build #54850 has finished for PR 12134 at commit e4f06f9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-04T16:10:05Z

Test build #54851 has finished for PR 12134 at commit 07264c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-04T16:12:37Z

Test build #54852 has finished for PR 12134 at commit c618a78.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-04-04T19:06:19Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala

@@ -201,6 +202,31 @@ class HiveQlSuite extends SparkFunSuite with BeforeAndAfterAll {
    assert(plan.children.head.asInstanceOf[Generate].generator.isInstanceOf[JsonTuple])
  }

+  test("transform query spec") {
+    val plan1 = parser.parsePlan("select transform(a, b) using 'func' as c, d from e")


What is the reason for changing the test? Is HiveQlSuite written in a different style?

: ) Let me import all the dsl from Catalyst. I have no clue why the test cases in sql/hive does not do it.

hvanhovell · 2016-04-04T19:09:53Z

@gatorsmile we are getting there. Two minor comments.

SparkQA · 2016-04-04T19:28:45Z

Test build #54868 has finished for PR 12134 at commit 4486c75.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class HiveQlSuite extends PlanTest

SparkQA · 2016-04-05T03:04:32Z

Test build #54919 has finished for PR 12134 at commit 249febf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-04-05T06:47:31Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala

@@ -351,7 +364,8 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with Logging {
          string(script),
          attributes,
          withFilter,
-          withScriptIOSchema(inRowFormat, recordWriter, outRowFormat, recordReader, schemaLess))
+          withScriptIOSchema(


Nit: one line (i.e. revert this).

Thanks! After we adding ctx, it has 102 characters. Thus, have to split it into two lines.

Ok, my bad :)

hvanhovell · 2016-04-05T06:49:26Z

LGTM - pending rebase/merge and jenkins.

# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandSuite.scala

SparkQA · 2016-04-05T07:15:08Z

Test build #54965 has finished for PR 12134 at commit 8a7fcf6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-04-05T07:19:09Z

retest this please

SparkQA · 2016-04-05T07:30:15Z

Test build #54966 has finished for PR 12134 at commit 8a7fcf6.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-05T09:13:56Z

Test build #2756 has finished for PR 12134 at commit 8a7fcf6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2016-04-05T09:18:12Z

Merging to master. Thanks!

gatorsmile and others added 30 commits November 13, 2015 14:50

Merge remote-tracking branch 'upstream/master'

01e4cdf

Merge remote-tracking branch 'upstream/master'

6835704

Merge remote-tracking branch 'upstream/master'

9180687

SPARK-11633

b38a21e

Merge remote-tracking branch 'upstream/master' into joinMakeCopy

d2b84af

Merge remote-tracking branch 'upstream/master'

fda8025

Merge branch 'master' of https://github.com/gatorsmile/spark

ac0dccd

Merge remote-tracking branch 'upstream/master'

6e0018b

converge

0546772

converge

b37a64f

Merge remote-tracking branch 'upstream/master'

c2a872c

Merge remote-tracking branch 'upstream/master'

ab6dbd7

Merge remote-tracking branch 'upstream/master'

4276356

Merge remote-tracking branch 'upstream/master'

2dab708

Merge remote-tracking branch 'upstream/master'

0458770

Merge remote-tracking branch 'upstream/master'

1debdfa

Merge remote-tracking branch 'upstream/master'

763706d

Merge remote-tracking branch 'upstream/master'

4de6ec1

Merge remote-tracking branch 'upstream/master'

9422a4f

Merge remote-tracking branch 'upstream/master'

52bdf48

Merge remote-tracking branch 'upstream/master'

1e95df3

Merge remote-tracking branch 'upstream/master'

fab24cf

Merge remote-tracking branch 'upstream/master'

8b2e33b

Merge remote-tracking branch 'upstream/master'

2ee1876

Merge remote-tracking branch 'upstream/master'

b9f0090

Merge remote-tracking branch 'upstream/master'

ade6f7e

Merge remote-tracking branch 'upstream/master'

9fd63d2

Merge remote-tracking branch 'upstream/master'

5199d49

Merge remote-tracking branch 'upstream/master'

404214c

Merge remote-tracking branch 'upstream/master'

c001dd9

gatorsmile added 2 commits April 3, 2016 23:44

address comments.

c7c9461

address comments.

e69a7e8

hvanhovell reviewed Apr 4, 2016
View reviewed changes

gatorsmile added 3 commits April 4, 2016 07:26

revert it back.

e4f06f9

revert it back.

07264c0

revert it back.

c618a78

move test cases.

4486c75

hvanhovell reviewed Apr 4, 2016
View reviewed changes

address comments.

249febf

gatorsmile mentioned this pull request Apr 5, 2016

[SPARK-14362] [SPARK-14406] [SQL] DDL Native Support: Drop View and Drop Table #12146

Closed

hvanhovell reviewed Apr 5, 2016
View reviewed changes

gatorsmile added 2 commits April 5, 2016 00:02

Merge remote-tracking branch 'upstream/master'

65bd090

Merge branch 'hiveParserCommand' into hiveParserCommandNew

8a7fcf6

# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandSuite.scala

asfgit closed this in 7807173 Apr 5, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-14349] [SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. #12134

[SPARK-14349] [SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. #12134

gatorsmile commented Apr 3, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

hvanhovell Apr 4, 2016

gatorsmile Apr 4, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

hvanhovell Apr 4, 2016

gatorsmile Apr 5, 2016

hvanhovell commented Apr 4, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 5, 2016

hvanhovell Apr 5, 2016

gatorsmile Apr 5, 2016

hvanhovell Apr 5, 2016

gatorsmile Apr 5, 2016

hvanhovell commented Apr 5, 2016

SparkQA commented Apr 5, 2016

gatorsmile commented Apr 5, 2016

SparkQA commented Apr 5, 2016

SparkQA commented Apr 5, 2016

hvanhovell commented Apr 5, 2016

[SPARK-14349] [SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. #12134

[SPARK-14349] [SQL] Issue Error Messages for Unsupported Operators/DML/DDL in SQL Context. #12134

Conversation

gatorsmile commented Apr 3, 2016

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell commented Apr 4, 2016

SparkQA commented Apr 4, 2016

SparkQA commented Apr 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hvanhovell commented Apr 5, 2016

SparkQA commented Apr 5, 2016

gatorsmile commented Apr 5, 2016

SparkQA commented Apr 5, 2016

SparkQA commented Apr 5, 2016

hvanhovell commented Apr 5, 2016