Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-6542][SQL] add CreateStruct #5195

Closed
wants to merge 5 commits into from
Closed

[SPARK-6542][SQL] add CreateStruct #5195

wants to merge 5 commits into from

Conversation

mengxr
Copy link
Contributor

@mengxr mengxr commented Mar 26, 2015

Similar to CreateArray, we can add CreateStruct to create nested columns. @marmbrus

@mengxr mengxr mentioned this pull request Mar 26, 2015
Closed
@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29197 has started for PR 5195 at commit 85f3106.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29197 has finished for PR 5195 at commit 85f3106.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UnregisterApplication(appId: String)
    • class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer]
    • case class Data(weight: Double, mu: Vector, sigma: Matrix)
    • case class CreateStruct(children: Seq[Expression]) extends Expression

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29197/
Test FAILed.

@liancheng
Copy link
Contributor

The build failure was caused by codegen. Some test suite executed earlier enabled codegen, thus CreateStruct.dataType got called somewhere and caused the build failure. When codegen is disabled, CreateStruct.dataType is not touched.

@mengxr mengxr changed the title [SPARK-6542][SQL] add CreateStruct [WIP][SPARK-6542][SQL] add CreateStruct Mar 26, 2015
@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29252 has started for PR 5195 at commit 85dd559.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Mar 26, 2015

Test build #29252 has finished for PR 5195 at commit 85dd559.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class CreateStruct(children: Seq[NamedExpression]) extends Expression

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29252/
Test FAILed.

@mengxr mengxr changed the title [WIP][SPARK-6542][SQL] add CreateStruct [SPARK-6542][SQL] add CreateStruct Mar 31, 2015
@SparkQA
Copy link

SparkQA commented Mar 31, 2015

Test build #29465 has started for PR 5195 at commit ae7ac3e.

@SparkQA
Copy link

SparkQA commented Mar 31, 2015

Test build #29468 has started for PR 5195 at commit 3795c57.

@SparkQA
Copy link

SparkQA commented Mar 31, 2015

Test build #29465 has finished for PR 5195 at commit ae7ac3e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class CreateStruct(children: Seq[NamedExpression]) extends Expression
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29465/
Test PASSed.

@asfgit asfgit closed this in a05835b Mar 31, 2015
@liancheng
Copy link
Contributor

Merged to master, thanks!

@SparkQA
Copy link

SparkQA commented Mar 31, 2015

Test build #29468 has finished for PR 5195 at commit 3795c57.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class CreateStruct(children: Seq[NamedExpression]) extends Expression
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29468/
Test PASSed.

marmbrus
marmbrus reviewed Mar 31, 2015
View reviewed changes
...yst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala
@@ -1081,3 +1087,14 @@ class ExpressionEvaluationSuite extends FunSuite {
checkEvaluation(~c1, -2, row)
}
}

// TODO: Make the tests work with codegen.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a JIRA for this TODO?

asfgit pushed a commit that referenced this pull request Apr 13, 2015
@mengxr
[SPARK-5885][MLLIB] Add VectorAssembler as a feature transformer
9294044 
VectorAssembler merges multiple columns into a vector column. This PR contains content from #5195.

~~carry ML attributes~~ (moved to a follow-up PR)

Author: Xiangrui Meng <[email protected]>

Closes #5196 from mengxr/SPARK-5885 and squashes the following commits:

a52b101 [Xiangrui Meng] recognize more types
35daac2 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-5885
bb5e64b [Xiangrui Meng] add TODO for null
976a3d6 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-5885
0859311 [Xiangrui Meng] Revert "add CreateStruct"
29fb6ac [Xiangrui Meng] use CreateStruct
adb71c4 [Xiangrui Meng] Merge branch 'SPARK-6542' into SPARK-5885
85f3106 [Xiangrui Meng] add CreateStruct
4ff16ce [Xiangrui Meng] add VectorAssembler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mengxr @SparkQA @AmplabJenkins @liancheng @marmbrus