Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5554] [SQL] [PySpark] add more tests for DataFrame Python API #4331

Closed
wants to merge 10 commits into from
Closed

[SPARK-5554] [SQL] [PySpark] add more tests for DataFrame Python API #4331

wants to merge 10 commits into from

Conversation

davies
Copy link
Contributor

@davies davies commented Feb 3, 2015

Add more tests and docs for DataFrame Python API, improve test coverage, fix bugs.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26635 has started for PR 4331 at commit 3ab2661.

  • This patch merges cleanly.

@davies davies changed the title [SPARK-5554] [SQL] [PySpark] add more tests for DataFrame [SPARK-5554] [SQL] [PySpark] add more tests for DataFrame Python API Feb 3, 2015
Davies Liu added 3 commits February 3, 2015 01:09
fix docs
6040ba7
Merge branch 'master' of github.com:apache/spark into fix_df
9ab78b4 
Conflicts:
	sql/core/src/main/scala/org/apache/spark/sql/Column.scala
add sql_test.py in run_tests
78ebcfa
@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26643 has started for PR 4331 at commit 9ab78b4.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26643 has finished for PR 4331 at commit 9ab78b4.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26643/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26646 has started for PR 4331 at commit 78ebcfa.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26646 has finished for PR 4331 at commit 78ebcfa.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Dsl(object):
    • class ExamplePointUDT(UserDefinedType):
    • class SQLTests(ReusedPySparkTestCase):

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26646/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26649 has started for PR 4331 at commit 35ccb9f.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26635 has finished for PR 4331 at commit 3ab2661.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Dsl(object):
    • class ExamplePointUDT(UserDefinedType):
    • class SQLTests(ReusedPySparkTestCase):

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26635/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #575 has started for PR 4331 at commit 35ccb9f.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26649 has finished for PR 4331 at commit 35ccb9f.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26649/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #575 has finished for PR 4331 at commit 35ccb9f.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26661 has started for PR 4331 at commit 8dd19a9.

  • This patch merges cleanly.

rxin
rxin reviewed Feb 3, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/Column.scala
@@ -496,8 +496,8 @@ trait Column extends DataFrame {
* @param startPos expression for the starting position.
* @param len expression for the length of the substring.
*/
def substr(startPos: Column, len: Column): Column = {
new IncomputableColumn(Substring(expr, startPos.expr, len.expr))
def substr(startPos: Column, len: Column): Column = constructColumn(null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change breaks this pr : #4332

You need to use

exprToColumn(Substring(expr, startPos.expr, len.expr), computable = false)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why computable should be false?

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26661 has finished for PR 4331 at commit 8dd19a9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Dsl(object):
    • class ExamplePointUDT(UserDefinedType):
    • class SQLTests(ReusedPySparkTestCase):

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26661/
Test PASSed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26671 has started for PR 4331 at commit 83c92fe.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26672 has started for PR 4331 at commit 467332c.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26671 has finished for PR 4331 at commit 83c92fe.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26671/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26672 has finished for PR 4331 at commit 467332c.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Dsl(object):
    • class ExamplePointUDT(UserDefinedType):
    • class SQLTests(ReusedPySparkTestCase):

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26672/
Test FAILed.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26679 has started for PR 4331 at commit dd9919f.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 3, 2015

Test build #26679 has finished for PR 4331 at commit dd9919f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Dsl(object):
    • class ExamplePointUDT(UserDefinedType):
    • class SQLTests(ReusedPySparkTestCase):

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26679/
Test PASSed.

@rxin
Copy link
Contributor

rxin commented Feb 4, 2015

Thanks. Merging in master.

@asfgit asfgit closed this in 068c0e2 Feb 4, 2015
asfgit pushed a commit that referenced this pull request Feb 4, 2015
@rxin
[SPARK-5554] [SQL] [PySpark] add more tests for DataFrame Python API
4640623 
Add more tests and docs for DataFrame Python API, improve test coverage, fix bugs.

Author: Davies Liu <[email protected]>

Closes #4331 from davies/fix_df and squashes the following commits:

dd9919f [Davies Liu] fix tests
467332c [Davies Liu] support string in cast()
83c92fe [Davies Liu] address comments
c052f6f [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_df
8dd19a9 [Davies Liu] fix tests in python 2.6
35ccb9f [Davies Liu] fix build
78ebcfa [Davies Liu] add sql_test.py in run_tests
9ab78b4 [Davies Liu] Merge branch 'master' of github.com:apache/spark into fix_df
6040ba7 [Davies Liu] fix docs
3ab2661 [Davies Liu] add more tests for DataFrame

(cherry picked from commit 068c0e2)
Signed-off-by: Reynold Xin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@davies @SparkQA @AmplabJenkins @rxin