Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5709] [SQL] Add EXPLAIN support in DataFrame API for debugging purpose #4496

Closed
wants to merge 1 commit into from
Closed

[SPARK-5709] [SQL] Add EXPLAIN support in DataFrame API for debugging purpose #4496

wants to merge 1 commit into from

Conversation

chenghao-intel
Copy link
Contributor

No description provided.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27181 has started for PR 4496 at commit c2d5338.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27181 has finished for PR 4496 at commit c2d5338.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27181/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27184 has started for PR 4496 at commit 5d92e1b.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27184 has finished for PR 4496 at commit 5d92e1b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27184/
Test FAILed.

rxin
rxin reviewed Feb 10, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
@@ -116,6 +116,9 @@ trait DataFrame extends RDDApi[Row] {
/** Prints the schema to the console in a nice tree format. */
def printSchema(): Unit

/** Returns the plan information as string array for debugging purpose. */
def explain(extended: Boolean): Array[String]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems to me it'd make more sense for explain to just print something to the console by default, isn't it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I agree, I will update the code.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27205 has started for PR 4496 at commit 013b05a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 10, 2015

Test build #27205 has finished for PR 4496 at commit 013b05a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27205/
Test PASSed.

rxin
rxin reviewed Feb 10, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
@@ -116,6 +116,9 @@ trait DataFrame extends RDDApi[Row] {
/** Prints the schema to the console in a nice tree format. */
def printSchema(): Unit

/** Prints the plan to the console for debugging purpose. */
def printPlan(extended: Boolean): Unit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought explain was a pretty good name actually :)

@marmbrus what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to explain. Can we have a default that sets extended to false as well.

@SparkQA
Copy link

SparkQA commented Feb 11, 2015

Test build #27256 has started for PR 4496 at commit 44ddb78.

  • This patch does not merge cleanly.

@marmbrus
Copy link
Contributor

mind rebasing quickly? we are going to cut a snapshot release soon.

marmbrus
marmbrus reviewed Feb 11, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/Column.scala
@@ -600,6 +600,15 @@ trait Column extends DataFrame {
def desc: Column = exprToColumn(SortOrder(expr, Descending), computable = false)

def asc: Column = exprToColumn(SortOrder(expr, Ascending), computable = false)

override def explain(extended: Boolean): Unit = {
val e = this.expr
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd use prettyString

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually pretty string unless extended probably

@chenghao-intel
Copy link
Contributor Author

yes, I am working on it now.

marmbrus
marmbrus reviewed Feb 11, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/Column.scala

override def explain(extended: Boolean): Unit = {
val e = this.expr
if (e == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can never be null.

@SparkQA
Copy link

SparkQA commented Feb 11, 2015

Test build #27262 has started for PR 4496 at commit 552aa58.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 11, 2015

Test build #27256 has finished for PR 4496 at commit 44ddb78.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27256/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 11, 2015

Test build #27262 has finished for PR 4496 at commit 552aa58.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27262/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 11, 2015
@chenghao-intel @marmbrus
[SPARK-5709] [SQL] Add EXPLAIN support in DataFrame API for debugging…
7fa0d5f 
… purpose

Author: Cheng Hao <[email protected]>

Closes #4496 from chenghao-intel/df_explain and squashes the following commits:

552aa58 [Cheng Hao] Add explain support for DF

(cherry picked from commit 45df77b)
Signed-off-by: Michael Armbrust <[email protected]>
@asfgit asfgit closed this in 45df77b Feb 11, 2015
@chenghao-intel chenghao-intel deleted the df_explain branch July 2, 2015 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@chenghao-intel @SparkQA @AmplabJenkins @marmbrus @rxin