[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

liancheng · 2014-07-14T06:46:17Z

JIRA issue:

Main: SPARK-2410
Related: SPARK-2678

Cherry picked the Hive Thrift/JDBC server from branch-1.0-jdbc.

(Thanks @chenghao-intel for his initial contribution of the Spark SQL CLI.)

TODO

Use spark-submit to launch the server, the CLI and beeline
Migration guideline draft for Shark users

Hit by a bug in SparkSubmitArguments while working on this PR: all application options that are recognized by SparkSubmitArguments are stolen as SparkSubmit options. For example:

$ spark-submit --class org.apache.hive.beeline.BeeLine spark-internal --help

This actually shows usage information of SparkSubmit rather than BeeLine.

~~Fixed this bug here since the spark-internal related stuff also touches SparkSubmitArguments and I'd like to avoid conflict.~~

UPDATE The bug mentioned above is now tracked by SPARK-2678. Decided to revert changes to this bug since it involves more subtle considerations and worth a separate PR.

SparkQA · 2014-07-14T06:47:41Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16615/consoleFull

SparkQA · 2014-07-14T06:47:46Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16615/consoleFull

liancheng · 2014-07-14T07:14:16Z

Hey @pwendell, would you mind help to retest this PR and review my changes to the POM and SBT files? Not 100% sure about the correctness. Thanks!

CodingCat · 2014-07-16T04:37:44Z

sbin/start-thriftserver.sh

+FWDIR="$(cd `dirname $0`/..; pwd)"
+
+CLASS="org.apache.spark.sql.hive.thriftserver.HiveThriftServer2"
+$FWDIR/bin/spark-class $CLASS $@


ah, some discussion from the JIRA: here we can use spark-submit to launch the server to enable the user to set cores/memory used by thriftserver.....

Thanks, I've noticed the discussion. Marked this PR as WIP, will update soon.

Checkout spark-shell for an example of a user-facing script that triages options to spark-submit.

SparkQA · 2014-07-20T14:43:10Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16871/consoleFull

SparkQA · 2014-07-20T14:43:16Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16871/consoleFull

SparkQA · 2014-07-20T15:28:07Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16873/consoleFull

SparkQA · 2014-07-20T15:28:54Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16873/consoleFull

SparkQA · 2014-07-20T15:53:10Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16874/consoleFull

SparkQA · 2014-07-20T15:53:47Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16874/consoleFull

SparkQA · 2014-07-20T16:13:04Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16875/consoleFull

SparkQA · 2014-07-20T17:52:24Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16875/consoleFull

SparkQA · 2014-07-21T03:23:10Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16890/consoleFull

pwendell · 2014-07-21T04:02:41Z

assembly/pom.xml

@@ -162,6 +162,11 @@
          <artifactId>spark-hive_${scala.binary.version}</artifactId>
          <version>${project.version}</version>
        </dependency>
+        <dependency>
+          <groupId>org.apache.spark</groupId>
+          <artifactId>spark-hive-thriftserver_${scala.binary.version}</artifactId>


Rather than have this included in the hive profile, we should make a hive-thriftserver profile. I could imagine users who want to build with support for reading Hive data, but don't care about running a JDBC server.

Also - if you do this you should modify the relevant documentation, the create-release.sh file, the dev/run-tests file, and the dev/scalastyle file (basically do a grep for -Phive in the codebase).

SparkQA · 2014-07-21T04:58:12Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class HiveThriftServer2(hiveContext: HiveContext)
class SparkSQLCLIDriver extends CliDriver with Logging {
class SparkSQLCLIService(hiveContext: HiveContext)
class SparkSQLDriver(val context: HiveContext = SparkSQLEnv.hiveContext)
class SparkSQLSessionManager(hiveContext: HiveContext)
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16890/consoleFull

SparkQA · 2014-07-22T09:13:21Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16959/consoleFull

SparkQA · 2014-07-22T10:21:01Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16959/consoleFull

SparkQA · 2014-07-22T14:48:22Z

QA tests have started for PR 1399. This patch DID NOT merge cleanly!
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16967/consoleFull

SparkQA · 2014-07-22T15:18:16Z

QA tests have started for PR 1399. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16968/consoleFull

liancheng · 2014-07-22T15:28:12Z

Hey @pwendell, thanks for the detailed comments and review. I believe the last a few commits have addressed all issues brought up. I rebased this PR to the most recent master. The bug found in SparkSubmitArguments was also fixed.

@rxin @marmbrus Would you please help review the Hive compatibility and Shark migration guide draft sections of the updated documents? Thanks in advance!

SparkQA · 2014-07-22T16:01:30Z

QA results for PR 1399:
- This patch FAILED unit tests.

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16967/consoleFull

SparkQA · 2014-07-22T16:30:36Z

QA results for PR 1399:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
class SparkSQLOperationManager(hiveContext: HiveContext) extends OperationManager with Logging {

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16968/consoleFull

liancheng · 2014-07-22T23:21:35Z

The build failure was due to mima configuration, just disabled mima for hive-thriftserver.

This reverts commit 06dc0d2. #1399 is making Jenkins fail. We should investigate and put this back after its passing tests. Author: Michael Armbrust <[email protected]> Closes #1594 from marmbrus/revertJDBC and squashes the following commits: 59748da [Michael Armbrust] Revert "[SPARK-2410][SQL] Merging Hive Thrift/JDBC server"

marmbrus · 2014-07-25T22:40:40Z

Hey this was making Jenkins fail so I reverted it. We should investigate and try again.

liancheng · 2014-07-26T01:37:21Z

@marmbrus Sorry for the trouble. It's weird that the build suddenly breaks continuously after several heathy builds. Can't reproduce the failure locally right now, I'll merge PRs of those failed builds to this branch to see what happens. One suspicious reason jump into my head is HiveThrfitServer2Suite uses a hard coded port number (10000) and may cause collision with other parallel builds. Not familiar with our Jenkins setup, is it possible?

rxin · 2014-07-26T01:39:34Z

Is there a fixed port or something that we use? On Jenkins we run multiple tests at the same time so if you have a single fixed port, tests can fail.

liancheng · 2014-07-26T01:43:03Z

Ah, thanks, then I think this might be the problem.

liancheng · 2014-07-26T03:58:15Z

Opened #1600 to replace this PR. Hope Mr. Jenkins accepts it gently...

@pwendell

(This is a replacement of #1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <[email protected]> Closes #1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

… fix) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar. In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity. Author: Cheng Lian <[email protected]> Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits: 629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server

@pwendell

JIRA issue: - Main: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) - Related: [SPARK-2678](https://issues.apache.org/jira/browse/SPARK-2678) Cherry picked the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). (Thanks chenghao-intel for his initial contribution of the Spark SQL CLI.) TODO - [x] Use `spark-submit` to launch the server, the CLI and beeline - [x] Migration guideline draft for Shark users ---- Hit by a bug in `SparkSubmitArguments` while working on this PR: all application options that are recognized by `SparkSubmitArguments` are stolen as `SparkSubmit` options. For example: ```bash $ spark-submit --class org.apache.hive.beeline.BeeLine spark-internal --help ``` This actually shows usage information of `SparkSubmit` rather than `BeeLine`. ~~Fixed this bug here since the `spark-internal` related stuff also touches `SparkSubmitArguments` and I'd like to avoid conflict.~~ **UPDATE** The bug mentioned above is now tracked by [SPARK-2678](https://issues.apache.org/jira/browse/SPARK-2678). Decided to revert changes to this bug since it involves more subtle considerations and worth a separate PR. Author: Cheng Lian <[email protected]> Closes apache#1399 from liancheng/thriftserver and squashes the following commits: 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

This reverts commit 06dc0d2. apache#1399 is making Jenkins fail. We should investigate and put this back after its passing tests. Author: Michael Armbrust <[email protected]> Closes apache#1594 from marmbrus/revertJDBC and squashes the following commits: 59748da [Michael Armbrust] Revert "[SPARK-2410][SQL] Merging Hive Thrift/JDBC server"

@pwendell

(This is a replacement of apache#1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](apache#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <[email protected]> Closes apache#1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

… fix) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Another try for apache#1399 & apache#1600. Those two PR breaks Jenkins builds because we made a separate profile `hive-thriftserver` in sub-project `assembly`, but the `hive-thriftserver` module is defined outside the `hive-thriftserver` profile. Thus every time a pull request that doesn't touch SQL code will also execute test suites defined in `hive-thriftserver`, but tests fail because related .class files are not included in the assembly jar. In the most recent commit, module `hive-thriftserver` is moved into its own profile to fix this problem. All previous commits are squashed for clarity. Author: Cheng Lian <[email protected]> Closes apache#1620 from liancheng/jdbc-with-maven-fix and squashes the following commits: 629988e [Cheng Lian] Moved hive-thriftserver module definition into its own profile ec3c7a7 [Cheng Lian] Cherry picked the Hive Thrift server

@pwendell

(This is a replacement of #1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](apache/spark#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <[email protected]> Closes #1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

@pwendell

(This is a replacement of #1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](apache/spark#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <[email protected]> Closes #1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

@pwendell

(This is a replacement of #1399, trying to fix potential `HiveThriftServer2` port collision between parallel builds. Please refer to [these comments](apache/spark#1399 (comment)) for details.) JIRA issue: [SPARK-2410](https://issues.apache.org/jira/browse/SPARK-2410) Merging the Hive Thrift/JDBC server from [branch-1.0-jdbc](https://github.com/apache/spark/tree/branch-1.0-jdbc). Thanks chenghao-intel for his initial contribution of the Spark SQL CLI. Author: Cheng Lian <[email protected]> Closes #1600 from liancheng/jdbc and squashes the following commits: ac4618b [Cheng Lian] Uses random port for HiveThriftServer2 to avoid collision with parallel builds 090beea [Cheng Lian] Revert changes related to SPARK-2678, decided to move them to another PR 21c6cf4 [Cheng Lian] Updated Spark SQL programming guide docs fe0af31 [Cheng Lian] Reordered spark-submit options in spark-shell[.cmd] 199e3fb [Cheng Lian] Disabled MIMA for hive-thriftserver 1083e9d [Cheng Lian] Fixed failed test suites 7db82a1 [Cheng Lian] Fixed spark-submit application options handling logic 9cc0f06 [Cheng Lian] Starts beeline with spark-submit cfcf461 [Cheng Lian] Updated documents and build scripts for the newly added hive-thriftserver profile 061880f [Cheng Lian] Addressed all comments by @pwendell 7755062 [Cheng Lian] Adapts test suites to spark-submit settings 40bafef [Cheng Lian] Fixed more license header issues e214aab [Cheng Lian] Added missing license headers b8905ba [Cheng Lian] Fixed minor issues in spark-sql and start-thriftserver.sh f975d22 [Cheng Lian] Updated docs for Hive compatibility and Shark migration guide draft 3ad4e75 [Cheng Lian] Starts spark-sql shell with spark-submit a5310d1 [Cheng Lian] Make HiveThriftServer2 play well with spark-submit 61f39f4 [Cheng Lian] Starts Hive Thrift server via spark-submit 2c4c539 [Cheng Lian] Cherry picked the Hive Thrift server

CodingCat reviewed Jul 16, 2014
View reviewed changes

liancheng changed the title ~~[SPARK-2410][SQL] Cherry picked Hive Thrift/JDBC server~~ [SPARK-2410][SQL][WIP] Cherry picked Hive Thrift/JDBC server Jul 16, 2014

pwendell reviewed Jul 21, 2014
View reviewed changes

liancheng mentioned this pull request Jul 26, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1600

Closed

liancheng mentioned this pull request Jul 28, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server (with Maven profile fix) #1620

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

liancheng commented Jul 14, 2014

SparkQA commented Jul 14, 2014

SparkQA commented Jul 14, 2014

liancheng commented Jul 14, 2014

CodingCat Jul 16, 2014

liancheng Jul 16, 2014

pwendell Jul 17, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 21, 2014

pwendell Jul 21, 2014

pwendell Jul 21, 2014

liancheng Jul 21, 2014

SparkQA commented Jul 21, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

liancheng commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

liancheng commented Jul 22, 2014

marmbrus commented Jul 25, 2014

liancheng commented Jul 26, 2014

rxin commented Jul 26, 2014

liancheng commented Jul 26, 2014

liancheng commented Jul 26, 2014

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

[SPARK-2410][SQL] Merging Hive Thrift/JDBC server #1399

Conversation

liancheng commented Jul 14, 2014

SparkQA commented Jul 14, 2014

SparkQA commented Jul 14, 2014

liancheng commented Jul 14, 2014

CodingCat Jul 16, 2014

Choose a reason for hiding this comment

liancheng Jul 16, 2014

Choose a reason for hiding this comment

pwendell Jul 17, 2014

Choose a reason for hiding this comment

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 20, 2014

SparkQA commented Jul 21, 2014

pwendell Jul 21, 2014

Choose a reason for hiding this comment

pwendell Jul 21, 2014

Choose a reason for hiding this comment

liancheng Jul 21, 2014

Choose a reason for hiding this comment

SparkQA commented Jul 21, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

liancheng commented Jul 22, 2014

SparkQA commented Jul 22, 2014

SparkQA commented Jul 22, 2014

liancheng commented Jul 22, 2014

marmbrus commented Jul 25, 2014

liancheng commented Jul 26, 2014

rxin commented Jul 26, 2014

liancheng commented Jul 26, 2014

liancheng commented Jul 26, 2014