[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

liancheng · 2014-08-19T16:01:59Z

This PR fixes two issues:

Fixes wrongly quoted command line option in HiveThriftServer2Suite that makes test cases hang until timeout.
Asks dev/run-test to run Spark SQL tests when bin/spark-sql and/or sbin/start-thriftserver.sh are modified.

…tserver.sh

SparkQA · 2014-08-19T16:05:26Z

QA tests have started for PR 2036 at commit 26b82a0.

This patch merges cleanly.

SparkQA · 2014-08-19T17:20:32Z

QA tests have finished for PR 2036 at commit 26b82a0.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-08-19T21:59:00Z

This still appears to be failing on Jenkins. Any ideas?

chutium · 2014-08-19T23:09:44Z

hmm... jenkins is not working again?... https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18886/consoleFull

marmbrus · 2014-08-19T23:14:33Z

@chutium, that link appears to be for a different PR (#2044), but yeah it looks like one of the workspaces did not get cleaned up

liancheng · 2014-08-19T23:32:48Z

It's probably because of too small a timeout value in CliSuite.

marmbrus · 2014-08-19T23:35:13Z

Actually I get the same error locally... I see this if I add more debug logging:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:301)
    at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:271)
    at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
    at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
    at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:359)
    at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:359)
    at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
    at org.apache.spark.sql.SchemaRDD.<init>(SchemaRDD.scala:103)
    at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:58)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:291)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:226)
    at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Are you gonna have time to work on this soon or are you traveling? I might take a look.

liancheng · 2014-08-20T00:03:26Z

I'm heading to the airport, should have a couple hours there but not guaranteed. So yes please have a look at it if you have time, thanks!

liancheng · 2014-08-20T01:56:31Z

My fault, forgot to fix the same quotation issue in CliSuite. Let's see whether this one can be accepted by Jenkins.

SparkQA · 2014-08-20T02:00:22Z

QA tests have started for PR 2036 at commit f38c4eb.

This patch merges cleanly.

SparkQA · 2014-08-20T02:00:29Z

QA tests have finished for PR 2036 at commit f38c4eb.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-08-20T02:01:15Z

Jenkins, retest this please.

SparkQA · 2014-08-20T02:05:20Z

QA tests have started for PR 2036 at commit f38c4eb.

This patch merges cleanly.

SparkQA · 2014-08-20T02:05:25Z

QA tests have finished for PR 2036 at commit f38c4eb.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-08-20T04:02:30Z

Seems that some garbage files are left on Jenkins, causing build failures:

=========================================================================
Running Apache RAT checks
=========================================================================
Could not find Apache license headers in the following files:
 !????? /home/jenkins/workspace/SparkPullRequestBuilder/mllib/checkpoint/.temp.crc
 !????? /home/jenkins/workspace/SparkPullRequestBuilder/mllib/checkpoint/temp

liancheng · 2014-08-20T04:15:40Z

Checked Jenkins, the workspace is clean now. Let's try again.

liancheng · 2014-08-20T04:15:50Z

Jenkins, retest this please.

SparkQA · 2014-08-20T04:20:27Z

QA tests have started for PR 2036 at commit f38c4eb.

This patch merges cleanly.

SparkQA · 2014-08-20T04:20:33Z

QA tests have finished for PR 2036 at commit f38c4eb.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

liancheng · 2014-08-20T04:28:12Z

Hmm... The workspace is polluted again :(

rxin · 2014-08-20T05:44:09Z

Jenkins, retest this please.

chenghao-intel · 2014-08-20T08:40:06Z

The unit test failure probably due to the assembly jar with duplicated netty jars (netty.3.2.2.Final.jar & netty.3.6.6.Final.jar, the class NioWorker with different method signatures), I've created another PR #2035 to fix this.

And of course the " should be removed as @liancheng did already.

marmbrus · 2014-08-20T18:16:51Z

test this please

SparkQA · 2014-08-20T18:20:31Z

QA tests have started for PR 2036 at commit f38c4eb.

This patch merges cleanly.

SparkQA · 2014-08-20T19:41:20Z

QA tests have finished for PR 2036 at commit f38c4eb.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2014-08-20T19:55:31Z

...iftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suite.scala

-        if (VERBOSE && stderr.length > 0) {
-          println(stderr)
-        }
+        print(stdout)


Do we really want to print this always? Perhaps we can just print when there is a failure like we do in HiveContext

Fixed this in #1856, which provides a faster and robuster version of these test suites, and is much more quiet when everything's OK. We can take care of that one later though.

marmbrus · 2014-08-20T19:55:57Z

Yay! SQL tests are passing again! I'm going to merge this into master and 1.1.

This PR fixes two issues: 1. Fixes wrongly quoted command line option in `HiveThriftServer2Suite` that makes test cases hang until timeout. 1. Asks `dev/run-test` to run Spark SQL tests when `bin/spark-sql` and/or `sbin/start-thriftserver.sh` are modified. Author: Cheng Lian <[email protected]> Closes #2036 from liancheng/fix-thriftserver-test and squashes the following commits: f38c4eb [Cheng Lian] Fixed the same quotation issue in CliSuite 26b82a0 [Cheng Lian] Run SQL tests when dff contains bin/spark-sql and/or sbin/start-thriftserver.sh a87f83d [Cheng Lian] Extended timeout e5aa31a [Cheng Lian] Fixed metastore JDBC URI quotation (cherry picked from commit cf46e72) Signed-off-by: Michael Armbrust <[email protected]>

This PR fixes two issues: 1. Fixes wrongly quoted command line option in `HiveThriftServer2Suite` that makes test cases hang until timeout. 1. Asks `dev/run-test` to run Spark SQL tests when `bin/spark-sql` and/or `sbin/start-thriftserver.sh` are modified. Author: Cheng Lian <[email protected]> Closes apache#2036 from liancheng/fix-thriftserver-test and squashes the following commits: f38c4eb [Cheng Lian] Fixed the same quotation issue in CliSuite 26b82a0 [Cheng Lian] Run SQL tests when dff contains bin/spark-sql and/or sbin/start-thriftserver.sh a87f83d [Cheng Lian] Extended timeout e5aa31a [Cheng Lian] Fixed metastore JDBC URI quotation

… expression (apache#2036) * fix: Avoid redundant array transform * add test * More --------- Co-authored-by: Liang-Chi Hsieh <[email protected]>

liancheng added 3 commits August 19, 2014 23:22

Fixed metastore JDBC URI quotation

e5aa31a

Extended timeout

a87f83d

Run SQL tests when dff contains bin/spark-sql and/or sbin/start-thrif…

26b82a0

…tserver.sh

Fixed the same quotation issue in CliSuite

f38c4eb

scwf mentioned this pull request Aug 20, 2014

[SQL] add note of use synchronizedMap in SQLConf #1996

Closed

marmbrus reviewed Aug 20, 2014
View reviewed changes

asfgit closed this in cf46e72 Aug 20, 2014

liancheng deleted the fix-thriftserver-test branch September 24, 2014 00:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

liancheng commented Aug 19, 2014

SparkQA commented Aug 19, 2014

SparkQA commented Aug 19, 2014

marmbrus commented Aug 19, 2014

chutium commented Aug 19, 2014

marmbrus commented Aug 19, 2014

liancheng commented Aug 19, 2014

marmbrus commented Aug 19, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

rxin commented Aug 20, 2014

chenghao-intel commented Aug 20, 2014

marmbrus commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

marmbrus Aug 20, 2014

liancheng Aug 25, 2014

marmbrus commented Aug 20, 2014

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

[SPARK-3126][SPARK-3127][SQL] Fixed HiveThriftServer2Suite #2036

Conversation

liancheng commented Aug 19, 2014

SparkQA commented Aug 19, 2014

SparkQA commented Aug 19, 2014

marmbrus commented Aug 19, 2014

chutium commented Aug 19, 2014

marmbrus commented Aug 19, 2014

liancheng commented Aug 19, 2014

marmbrus commented Aug 19, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

liancheng commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

liancheng commented Aug 20, 2014

rxin commented Aug 20, 2014

chenghao-intel commented Aug 20, 2014

marmbrus commented Aug 20, 2014

SparkQA commented Aug 20, 2014

SparkQA commented Aug 20, 2014

marmbrus Aug 20, 2014

Choose a reason for hiding this comment

liancheng Aug 25, 2014

Choose a reason for hiding this comment

marmbrus commented Aug 20, 2014