Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5573][SQL] Add explode to dataframes #4546

Closed
wants to merge 7 commits into from
Closed

[SPARK-5573][SQL] Add explode to dataframes #4546

wants to merge 7 commits into from

Conversation

marmbrus
Copy link
Contributor

No description provided.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27314 has started for PR 4546 at commit 950707a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27315 has started for PR 4546 at commit d633d01.

  • This patch merges cleanly.

rxin
rxin reviewed Feb 12, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/IncomputableColumn.scala
override def explode[A <: Product : TypeTag]
(input: Column*)(f: Row => TraversableOnce[A]): DataFrame = err()

/////////////////////////////////////////////////////////////////////////////
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while you are at it, this is indented wrong now

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27314 has finished for PR 4546 at commit 950707a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UserDefinedGenerator(
    • * case class Book(title: String, words: String)
    • * case class Word(word: String)

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27314/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27315 has finished for PR 4546 at commit d633d01.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UserDefinedGenerator(
    • * case class Book(title: String, words: String)
    • * case class Word(word: String)

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27315/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27318 has started for PR 4546 at commit a8d496c.

  • This patch merges cleanly.

rxin
rxin reviewed Feb 12, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala
f: A => TraversableOnce[B]): DataFrame = {
val dataType = ScalaReflection.schemaFor[B].dataType
val attributes = AttributeReference(outputColumn, dataType)() :: Nil
def rowFunction(row: Row) = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you define this as a val, i think it doesn't capture outer?

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27318 has finished for PR 4546 at commit a8d496c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UserDefinedGenerator(
    • trait DataFrame extends RDDApi[Row] with Serializable
    • * case class Book(title: String, words: String)
    • * case class Word(word: String)

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27318/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27340 has started for PR 4546 at commit eefd33a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 12, 2015

Test build #27340 has finished for PR 4546 at commit eefd33a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • case class UserDefinedGenerator(
    • * case class Book(title: String, words: String)
    • * case class Word(word: String)

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27340/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 12, 2015
@marmbrus
[SPARK-5573][SQL] Add explode to dataframes
c7eb9ee 
Author: Michael Armbrust <[email protected]>

Closes #4546 from marmbrus/explode and squashes the following commits:

eefd33a [Michael Armbrust] whitespace
a8d496c [Michael Armbrust] Merge remote-tracking branch 'apache/master' into explode
4af740e [Michael Armbrust] Merge remote-tracking branch 'origin/master' into explode
dc86a5c [Michael Armbrust] simple version
d633d01 [Michael Armbrust] add scala specific
950707a [Michael Armbrust] fix comments
ba8854c [Michael Armbrust] [SPARK-5573][SQL] Add explode to dataframes

(cherry picked from commit ee04a8b)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in ee04a8b Feb 12, 2015
@marmbrus marmbrus deleted the explode branch February 17, 2015 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@marmbrus @SparkQA @AmplabJenkins @rxin