-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-16727][SparkR] Fix expected test output of describe and summary functions #14357
Conversation
@junyangq Any idea how the tests were passing on Jenkins before this fix ? |
I think this is related to 142df48 |
Test build #62852 has finished for PR 14357 at commit
|
Hi, @shivaram. |
String columns works for |
@shivaram I'm also curious...The issue arised when I ran the test locally. |
> collect(describe(read.json('examples/src/main/resources/people.json')))
summary age name
1 count 2 3
2 mean 24.5 <NA>
3 stddev 7.7781745930520225 <NA>
4 min 19 Andy
5 max 30 Michael |
In fact, |
Hi, @junyangq . |
@dongjoon-hyun That makes sense. That's why I feel confused about the output on my local machine...
|
I see. Which version is it? |
I see. That seems 2.0 branch. |
Hi, @shivaram .
|
I think I ran the test under the master branch though... Let me double check :) |
Thank you, @junyangq . |
I merged the most recent master branch, rebuilt and installed the package, but the test failed at the same place. @dongjoon-hyun |
Oh, let me try this time. Thank you for double-checking, @junyangq . |
@junyangq . Currently, you are at the most recent master build, right? scala> spark.read.json("examples/src/main/resources/people.json").describe().show() |
Hmm... It doesn't show the name column either. |
At the most recent master build, I did the following things and all tests are passed.
|
I think you did something wrong. :) |
Actually, you can see the Jenkins log, too. There is no problem with the current R testsuite. FYI, I'm using JDK 1.8.0_102 and Jenkins is using JDK 1.7.x. |
Yeah sure, but just wondering if the clean and additional arguments something we should normally do? |
Never. I did that in order to make it sure. I don't do it frequently. |
If you use |
I see. Thank you for pointing that out :) I'll close the PR. |
It's my pleasure. See you later around Apache Spark. :) |
What changes were proposed in this pull request?
Fix expected test output of describe and summary functions. String columns are not summarized in the output.
How was this patch tested?
SparkR unit test.