[SPARK-2817] [SQL] add "show create table" support #1760

tianyi · 2014-08-04T05:15:36Z

In spark sql component, the "show create table" syntax had been disabled.
We thought it is a useful funciton to describe a hive table.

AmplabJenkins · 2014-08-04T05:17:41Z

Can one of the admins verify this patch?

chenghao-intel · 2014-08-04T05:19:22Z

Don't forget to commit the golden answer files.
You can get that by running "sbt/sbt -Phive=true hive/test".

tianyi · 2014-08-04T06:19:39Z

@chenghao-intel is these files all right?

chenghao-intel · 2014-08-04T07:21:36Z

Jenkins, test this please

tianyi · 2014-08-04T15:05:20Z

what's wrong with jenkins?

JoshRosen · 2014-08-04T15:28:24Z

Jenkins, test this please.

(Jenkins only listens to a pre-approved list of GitHub accounts).

liancheng · 2014-08-04T16:33:53Z

LGTM :)

marmbrus · 2014-08-04T21:19:50Z

test this please

marmbrus · 2014-08-04T21:20:10Z

Can you please add [SQL] to the title of any PRs that affect Spark SQL?

SparkQA · 2014-08-04T21:24:21Z

QA tests have started for PR 1760. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17876/consoleFull

SparkQA · 2014-08-04T22:47:59Z

QA results for PR 1760:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17876/consoleFull

tianyi · 2014-08-05T00:57:50Z

there are two problem:
1 the result of "show create table" contains some time properties, which could not be the same with the time of the case running. Do we have hadoop and hive environment on jenkins machine? Should I remove some golden files?

2 I could not find how to get the ${system:test.tmp.dir} variable on my laptop, I thought it could be a environment related problem, but it failed on jenkins too. I think I can fix this by modify the pom.xml in "sql/hive/".

marmbrus · 2014-08-05T01:03:15Z

We can add time / user specific properties to nonDeterministicLineIndicators

Regarding the test that requires a system property, we don't support that yet. I'd remove that test from the whitelist.

Another note: there is no hive/hadoop on jenkins so only tests with golden files will pass there.

…test which contains "${system:test.tmp.dir}"

tianyi · 2014-08-05T01:26:33Z

I modified the pom.xml in spark-hive module, to support ${system:test.tmp.dir}.

marmbrus · 2014-08-05T01:27:42Z

I don't think thats a maven thing is it? I think thats a hive substitution that we haven't implemented.

tianyi · 2014-08-05T01:43:33Z

Hi, Michael. I run the test on my laptop with this configuration, it pass the test which failed before. I also check that the apache-hive project create the same configuration in pom.xml.

marmbrus · 2014-08-05T01:46:08Z

Oh, I see, its a system property that is set by maven and then threaded though to hive. Thank you for explaining.

I'd still just skip this test as thats not going to work for sbt (which is what Jenkins uses), and I don't want to have to change all the build files for this small detail.

marmbrus · 2014-08-05T01:47:08Z

Another option would be to set this system property somewhere in the HiveComparisionTest instead of in the build file.

tianyi · 2014-08-06T10:41:31Z

how about adding another rule in rewritePaths funciton in TestHive.scala?

marmbrus · 2014-08-06T17:42:55Z

Yeah, but then we have to handle escaping and such ourselves and if there are other properties the function could get unwieldily. Is there a problem calling System.setProperty(...) in the constructor of TestHive?

tianyi · 2014-08-07T01:11:52Z

2 question:
Q1: There are few variable in hive test sql, like:
system:build.ivy.lib.dir
system:test.dfs.mkdir
system:test.src.data.dir
system:test.tmp.dir
system:xxx
should I add them all in TestHive ?
Q2: In Hive project, the "test.tmp.dir" had been set to "${project.build.directory}/tmp" , I check all the SystemProperties , the "user.dir" is the closest one to ${project.build.directory}. Should I find best value for other variable?

add "${system:test.tmp.dir}" support add "last_modified_by" to nonDeterministicLineIndicators in HiveComparisonTest

tianyi · 2014-08-07T03:50:25Z

I just add "${system:test.tmp.dir}" support for now.

…ide the rewritePaths

tianyi · 2014-08-08T05:45:01Z

Hi Michael , could you review these codes again?

marmbrus · 2014-08-11T17:14:47Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala

@@ -60,6 +60,8 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
  // without restarting the JVM.
  System.clearProperty("spark.hostPort")

+
+


Can you remove these spurious new line additions? Below as well.

…n the source tree, and also clean some empty line

marmbrus · 2014-08-12T03:27:08Z

ok to test

SparkQA · 2014-08-12T03:29:56Z

QA tests have started for PR 1760. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18355/consoleFull

SparkQA · 2014-08-12T03:33:20Z

QA results for PR 1760:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18355/consoleFull

SparkQA · 2014-08-12T05:29:51Z

QA tests have started for PR 1760. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18359/consoleFull

tianyi · 2014-08-12T05:30:28Z

I'm sorry for forgetting run the test yesterday.
this time, i passed all the test on my laptop except udf_unix_timestamp function, I guess it should be a enviorment problem.

liancheng · 2014-08-12T05:59:36Z

Be aware that the udf_unix_timestamp case is timezone sensitive. That's why we reset timezone to "America/Los_Angeles" in beforeAll. This may be related to your test failure.

SparkQA · 2014-08-12T06:34:37Z

QA results for PR 1760:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18359/consoleFull

tianyi · 2014-08-13T00:21:40Z

@marmbrus I think the test failed because some enviorment problem on jenkins host. could you run it again?

marmbrus · 2014-08-13T00:33:49Z

test this please

SparkQA · 2014-08-13T00:39:54Z

QA tests have started for PR 1760. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18399/consoleFull

SparkQA · 2014-08-13T01:49:28Z

QA results for PR 1760:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18399/consoleFull

tianyi · 2014-08-13T05:14:15Z

@marmbrus Could you ask someone help me to fix this error ?

Failed example:
srdd.collect()
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.6/doctest.py", line 1253, in run
compileflags, 1) in test.globs
File "<doctest pyspark.sql.SQLContext.inferSchema[6]>", line 1, in
srdd.collect()
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1613, in collect
rows = RDD.collect(self)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 724, in collect
bytesInJava = self._jrdd.collect().iterator()
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call
self.target_id, self.name)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
format(target_id, '.', name), value)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 35.0 failed 1 times, most recent failure: Lost task 1.0 in stage 35.0 (TID 72, localhost): java.lang.ClassCastException: java.lang.String cannot be cast to java.util.ArrayList
net.razorvine.pickle.objects.ArrayConstructor.construct(ArrayConstructor.java:33)
net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:617)
net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:170)
net.razorvine.pickle.Unpickler.load(Unpickler.java:84)
net.razorvine.pickle.Unpickler.loads(Unpickler.java:97)
......

marmbrus · 2014-08-13T22:31:10Z

Jenkins, test this please.

SparkQA · 2014-08-13T22:35:38Z

QA tests have started for PR 1760. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18487/consoleFull

SparkQA · 2014-08-13T23:48:01Z

QA results for PR 1760:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18487/consoleFull

marmbrus · 2014-08-13T23:50:17Z

This only failed streaming tests. I'm going to merge into master and 1.1.

Thanks!

In spark sql component, the "show create table" syntax had been disabled. We thought it is a useful funciton to describe a hive table. Author: tianyi <[email protected]> Author: tianyi <[email protected]> Author: tianyi <[email protected]> Closes #1760 from tianyi/spark-2817 and squashes the following commits: 7d28b15 [tianyi] [SPARK-2817] fix too short prefix problem cbffe8b [tianyi] [SPARK-2817] fix the case problem 565ec14 [tianyi] [SPARK-2817] fix the case problem 60d48a9 [tianyi] [SPARK-2817] use system temporary folder instead of temporary files in the source tree, and also clean some empty line dbe1031 [tianyi] [SPARK-2817] move some code out of function rewritePaths, as it may be called multiple times 9b2ba11 [tianyi] [SPARK-2817] fix the line length problem 9f97586 [tianyi] [SPARK-2817] remove test.tmp.dir from pom.xml bfc2999 [tianyi] [SPARK-2817] add "File.separator" support, create a "testTmpDir" outside the rewritePaths bde800a [tianyi] [SPARK-2817] add "${system:test.tmp.dir}" support add "last_modified_by" to nonDeterministicLineIndicators in HiveComparisonTest bb82726 [tianyi] [SPARK-2817] remove test which requires a system from the whitelist. bbf6b42 [tianyi] [SPARK-2817] add a systemProperties named "test.tmp.dir" to pass the test which contains "${system:test.tmp.dir}" a337bd6 [tianyi] [SPARK-2817] add "show create table" support a03db77 [tianyi] [SPARK-2817] add "show create table" support (cherry picked from commit 13f54e2) Signed-off-by: Michael Armbrust <[email protected]>

In spark sql component, the "show create table" syntax had been disabled. We thought it is a useful funciton to describe a hive table. Author: tianyi <[email protected]> Author: tianyi <[email protected]> Author: tianyi <[email protected]> Closes apache#1760 from tianyi/spark-2817 and squashes the following commits: 7d28b15 [tianyi] [SPARK-2817] fix too short prefix problem cbffe8b [tianyi] [SPARK-2817] fix the case problem 565ec14 [tianyi] [SPARK-2817] fix the case problem 60d48a9 [tianyi] [SPARK-2817] use system temporary folder instead of temporary files in the source tree, and also clean some empty line dbe1031 [tianyi] [SPARK-2817] move some code out of function rewritePaths, as it may be called multiple times 9b2ba11 [tianyi] [SPARK-2817] fix the line length problem 9f97586 [tianyi] [SPARK-2817] remove test.tmp.dir from pom.xml bfc2999 [tianyi] [SPARK-2817] add "File.separator" support, create a "testTmpDir" outside the rewritePaths bde800a [tianyi] [SPARK-2817] add "${system:test.tmp.dir}" support add "last_modified_by" to nonDeterministicLineIndicators in HiveComparisonTest bb82726 [tianyi] [SPARK-2817] remove test which requires a system from the whitelist. bbf6b42 [tianyi] [SPARK-2817] add a systemProperties named "test.tmp.dir" to pass the test which contains "${system:test.tmp.dir}" a337bd6 [tianyi] [SPARK-2817] add "show create table" support a03db77 [tianyi] [SPARK-2817] add "show create table" support

[SPARK-2817] add "show create table" support

a03db77

[SPARK-2817] add "show create table" support

a337bd6

tianyi changed the title ~~[SPARK-2817] add "show create table" support~~ [SPARK-2817] [SQL] add "show create table" support Aug 4, 2014

tianyi added 2 commits August 5, 2014 09:09

[SPARK-2817] add a systemProperties named "test.tmp.dir" to pass the …

bbf6b42

…test which contains "${system:test.tmp.dir}"

[SPARK-2817] remove test which requires a system from the whitelist.

bb82726

[SPARK-2817]

bde800a

add "${system:test.tmp.dir}" support add "last_modified_by" to nonDeterministicLineIndicators in HiveComparisonTest

[SPARK-2817] add "File.separator" support, create a "testTmpDir" outs…

bfc2999

…ide the rewritePaths

marmbrus reviewed Aug 11, 2014
View reviewed changes

[SPARK-2817] use system temporary folder instead of temporary files i…

60d48a9

…n the source tree, and also clean some empty line

tianyi added 3 commits August 12, 2014 12:07

[SPARK-2817] fix the case problem

565ec14

[SPARK-2817] fix the case problem

cbffe8b

[SPARK-2817] fix too short prefix problem

7d28b15

asfgit closed this in 13f54e2 Aug 13, 2014

liancheng mentioned this pull request Aug 16, 2014

[SPARK-3062] [SPARK-2970] [SQL] spark-sql script ends with IOException when EventLogging is enabled #1970

Closed

		@@ -60,6 +60,8 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
		// without restarting the JVM.
		System.clearProperty("spark.hostPort")

[SPARK-2817] [SQL] add "show create table" support #1760

[SPARK-2817] [SQL] add "show create table" support #1760

Conversation

tianyi commented Aug 4, 2014

AmplabJenkins commented Aug 4, 2014

chenghao-intel commented Aug 4, 2014

tianyi commented Aug 4, 2014

chenghao-intel commented Aug 4, 2014

tianyi commented Aug 4, 2014

JoshRosen commented Aug 4, 2014

liancheng commented Aug 4, 2014

marmbrus commented Aug 4, 2014

marmbrus commented Aug 4, 2014

SparkQA commented Aug 4, 2014

SparkQA commented Aug 4, 2014

tianyi commented Aug 5, 2014

marmbrus commented Aug 5, 2014

tianyi commented Aug 5, 2014

marmbrus commented Aug 5, 2014

tianyi commented Aug 5, 2014

marmbrus commented Aug 5, 2014

marmbrus commented Aug 5, 2014

tianyi commented Aug 6, 2014

marmbrus commented Aug 6, 2014

tianyi commented Aug 7, 2014

tianyi commented Aug 7, 2014

tianyi commented Aug 8, 2014

marmbrus Aug 11, 2014

Choose a reason for hiding this comment

marmbrus commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

SparkQA commented Aug 12, 2014

tianyi commented Aug 12, 2014

liancheng commented Aug 12, 2014

SparkQA commented Aug 12, 2014

tianyi commented Aug 13, 2014

marmbrus commented Aug 13, 2014

SparkQA commented Aug 13, 2014

SparkQA commented Aug 13, 2014

tianyi commented Aug 13, 2014

marmbrus commented Aug 13, 2014

SparkQA commented Aug 13, 2014

SparkQA commented Aug 13, 2014

marmbrus commented Aug 13, 2014