[SPARK-7133][SQL] Implement struct, array, and map field accessor #5744

cloud-fan · 2015-04-28T11:04:17Z

It's the first step: generalize UnresolvedGetField to support all map, struct, and array
TODO: add apply in Scala and __getitem__ in Python, and unify the getItem and getField methods to one single API(or should we keep them for compatibility?).

cloud-fan · 2015-04-28T11:06:32Z

cc @rxin @marmbrus , it should be easy to add apply method, so I'll do it after code review.

AmplabJenkins · 2015-04-28T11:07:09Z

Can one of the admins verify this patch?

liancheng · 2015-04-28T23:11:18Z

ok to test

AmplabJenkins · 2015-04-28T23:12:10Z

Merged build triggered.

AmplabJenkins · 2015-04-28T23:12:19Z

Merged build started.

SparkQA · 2015-04-28T23:13:42Z

Test build #31189 has started for PR 5744 at commit b9636f6.

SparkQA · 2015-04-29T00:27:37Z

Test build #31189 has finished for PR 5744 at commit b9636f6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedGetField(child: Expression, fieldExpr: Expression) extends UnaryExpression
- trait GetField extends UnaryExpression
- abstract class StructGetField extends GetField
- abstract class OrdinalGetField extends GetField
- case class SimpleStructGetField(child: Expression, field: StructField, ordinal: Int)
- case class ArrayStructGetField(
- case class ArrayOrdinalGetField(child: Expression, ordinal: Expression)
- case class MapOrdinalGetField(child: Expression, ordinal: Expression)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T00:27:41Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-29T00:27:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31189/
Test FAILed.

AmplabJenkins · 2015-04-29T01:42:09Z

Merged build triggered.

AmplabJenkins · 2015-04-29T01:42:19Z

Merged build started.

SparkQA · 2015-04-29T01:42:51Z

Test build #31204 has started for PR 5744 at commit 0ec5b96.

rxin · 2015-04-29T02:33:42Z

We should definitely keep the old function for compatibility.

rxin · 2015-04-29T02:35:52Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala

@@ -194,7 +194,7 @@ case class ResolvedStar(expressions: Seq[NamedExpression]) extends Star {
  override def toString: String = expressions.mkString("ResolvedStar(", ", ", ")")
 }

-case class UnresolvedGetField(child: Expression, fieldName: String) extends UnaryExpression {
+case class UnresolvedGetField(child: Expression, fieldExpr: Expression) extends UnaryExpression {


can you add some javadoc for this class. at the very least, we should say this can be used to get a field out of struct/map/array.

SparkQA · 2015-04-29T03:29:25Z

Test build #31204 has finished for PR 5744 at commit 0ec5b96.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedGetField(child: Expression, fieldExpr: Expression) extends UnaryExpression
- trait GetField extends UnaryExpression
- abstract class StructGetField extends GetField
- abstract class OrdinalGetField extends GetField
- case class SimpleStructGetField(child: Expression, field: StructField, ordinal: Int)
- case class ArrayStructGetField(
- case class ArrayOrdinalGetField(child: Expression, ordinal: Expression)
- case class MapOrdinalGetField(child: Expression, ordinal: Expression)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T03:29:30Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-29T03:29:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31204/
Test FAILed.

AmplabJenkins · 2015-04-29T04:07:12Z

Merged build triggered.

AmplabJenkins · 2015-04-29T04:07:20Z

Merged build started.

SparkQA · 2015-04-29T04:07:34Z

Test build #31236 has started for PR 5744 at commit d72e47e.

SparkQA · 2015-04-29T05:46:35Z

Test build #31236 has finished for PR 5744 at commit d72e47e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedGetField(child: Expression, fieldExpr: Expression) extends UnaryExpression
- trait GetField extends UnaryExpression
- abstract class StructGetField extends GetField
- abstract class OrdinalGetField extends GetField
- case class SimpleStructGetField(child: Expression, field: StructField, ordinal: Int)
- case class ArrayStructGetField(
- case class ArrayOrdinalGetField(child: Expression, ordinal: Expression)
- case class MapOrdinalGetField(child: Expression, ordinal: Expression)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-29T05:46:40Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-04-29T05:46:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31236/
Test FAILed.

cloud-fan · 2015-04-29T06:05:25Z

The failed test cases looks unrelated, retest?

rxin · 2015-04-29T06:06:52Z

Jenkins, retest this please.

rxin · 2015-04-29T06:06:58Z

Jenkins, add to whitelist.

AmplabJenkins · 2015-04-29T06:07:11Z

Merged build triggered.

AmplabJenkins · 2015-04-29T06:07:16Z

Merged build started.

SparkQA · 2015-04-29T06:08:55Z

Test build #31251 has started for PR 5744 at commit d72e47e.

AmplabJenkins · 2015-05-08T06:52:16Z

Merged build triggered.

AmplabJenkins · 2015-05-08T06:52:22Z

Merged build started.

SparkQA · 2015-05-08T06:53:11Z

Test build #32210 has started for PR 5744 at commit 7274041.

SparkQA · 2015-05-08T06:54:51Z

Test build #32210 has finished for PR 5744 at commit 7274041.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedExtractValue(child: Expression, extraction: Expression)
- trait ExtractValue extends UnaryExpression
- case class GetStructField(child: Expression, field: StructField, ordinal: Int)
- case class GetArrayStructFields(
- abstract class ExtractValueWithOrdinal extends ExtractValue
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, ordinal: Expression)
- case class CreateTableAsSelect(

AmplabJenkins · 2015-05-08T06:54:52Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-08T06:54:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32210/
Test FAILed.

AmplabJenkins · 2015-05-08T07:07:11Z

Merged build triggered.

AmplabJenkins · 2015-05-08T07:07:17Z

Merged build started.

SparkQA · 2015-05-08T07:09:04Z

Test build #32213 has started for PR 5744 at commit 715c589.

SparkQA · 2015-05-08T09:30:38Z

Test build #32213 has finished for PR 5744 at commit 715c589.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class UnresolvedExtractValue(child: Expression, extraction: Expression)
- trait ExtractValue extends UnaryExpression
- case class GetStructField(child: Expression, field: StructField, ordinal: Int)
- case class GetArrayStructFields(
- abstract class ExtractValueWithOrdinal extends ExtractValue
- case class GetArrayItem(child: Expression, ordinal: Expression)
- case class GetMapValue(child: Expression, ordinal: Expression)

AmplabJenkins · 2015-05-08T09:30:43Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-08T09:30:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32213/
Test PASSed.

marmbrus · 2015-05-08T18:50:10Z

Thanks! Merged to master and 1.4.

It's the first step: generalize UnresolvedGetField to support all map, struct, and array TODO: add `apply` in Scala and `__getitem__` in Python, and unify the `getItem` and `getField` methods to one single API(or should we keep them for compatibility?). Author: Wenchen Fan <[email protected]> Closes #5744 from cloud-fan/generalize and squashes the following commits: 715c589 [Wenchen Fan] address comments 7ea5b31 [Wenchen Fan] fix python test 4f0833a [Wenchen Fan] add python test f515d69 [Wenchen Fan] add apply method and test cases 8df6199 [Wenchen Fan] fix python test 239730c [Wenchen Fan] fix test compile 2a70526 [Wenchen Fan] use _bin_op in dataframe.py 6bf72bc [Wenchen Fan] address comments 3f880c3 [Wenchen Fan] add java doc ab35ab5 [Wenchen Fan] fix python test b5961a9 [Wenchen Fan] fix style c9d85f5 [Wenchen Fan] generalize UnresolvedGetField to support all map, struct, and array (cherry picked from commit 2d05f32) Signed-off-by: Michael Armbrust <[email protected]>

It's the first step: generalize UnresolvedGetField to support all map, struct, and array TODO: add `apply` in Scala and `__getitem__` in Python, and unify the `getItem` and `getField` methods to one single API(or should we keep them for compatibility?). Author: Wenchen Fan <[email protected]> Closes apache#5744 from cloud-fan/generalize and squashes the following commits: 715c589 [Wenchen Fan] address comments 7ea5b31 [Wenchen Fan] fix python test 4f0833a [Wenchen Fan] add python test f515d69 [Wenchen Fan] add apply method and test cases 8df6199 [Wenchen Fan] fix python test 239730c [Wenchen Fan] fix test compile 2a70526 [Wenchen Fan] use _bin_op in dataframe.py 6bf72bc [Wenchen Fan] address comments 3f880c3 [Wenchen Fan] add java doc ab35ab5 [Wenchen Fan] fix python test b5961a9 [Wenchen Fan] fix style c9d85f5 [Wenchen Fan] generalize UnresolvedGetField to support all map, struct, and array

rxin reviewed Apr 29, 2015
View reviewed changes

cloud-fan added 6 commits May 8, 2015 12:41

use _bin_op in dataframe.py

2a70526

fix test compile

239730c

fix python test

8df6199

add apply method and test cases

f515d69

add python test

4f0833a

fix python test

7ea5b31

cloud-fan force-pushed the generalize branch from 346ad13 to 7274041 Compare May 8, 2015 06:50

cloud-fan force-pushed the generalize branch from 7274041 to 69afdae Compare May 8, 2015 07:02

address comments

715c589

cloud-fan force-pushed the generalize branch from 69afdae to 715c589 Compare May 8, 2015 07:05

asfgit closed this in 2d05f32 May 8, 2015

cloud-fan deleted the generalize branch May 9, 2015 03:35

[SPARK-7133][SQL] Implement struct, array, and map field accessor #5744

[SPARK-7133][SQL] Implement struct, array, and map field accessor #5744

Conversation

cloud-fan commented Apr 28, 2015

cloud-fan commented Apr 28, 2015

AmplabJenkins commented Apr 28, 2015

liancheng commented Apr 28, 2015

AmplabJenkins commented Apr 28, 2015

AmplabJenkins commented Apr 28, 2015

SparkQA commented Apr 28, 2015

SparkQA commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

SparkQA commented Apr 29, 2015

rxin commented Apr 29, 2015

rxin Apr 29, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

SparkQA commented Apr 29, 2015

SparkQA commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

cloud-fan commented Apr 29, 2015

rxin commented Apr 29, 2015

rxin commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

AmplabJenkins commented Apr 29, 2015

SparkQA commented Apr 29, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

SparkQA commented May 8, 2015

SparkQA commented May 8, 2015

AmplabJenkins commented May 8, 2015

AmplabJenkins commented May 8, 2015

marmbrus commented May 8, 2015