[SPARK-1753] Warn about PySpark on YARN on Red Hat #682
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Changes look good to me. Do you know if its documented somewhere you have to build with maven to get pyspark on yarn? |
|
I took a quick pass at the latest docs, and it looks like for 1.0+ we only mention maven when we talk about building. I wonder if we should still document the requirement for building with maven for PySpark on YARN, however, since we can still build with sbt even though it's not documented. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14782/ |
| @@ -129,6 +128,10 @@ Java 8 tests are run when -Pjava8-tests profile is enabled, they will run in spi | |||
| For these tests to run your system must have a JDK 8 installation. | |||
| If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests. | |||
|
|
|||
| ## Building for PySpark on YARN ## | |||
There was a problem hiding this comment.
It would be good to create a JIRA number for this and then reference it in the documentation.
andrewor14
changed the title
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
This PR is subsumed by #701. Closing. |
…ng (apache#681)" (apache#682) This reverts commit 035f4e4.
…ng (apache#681)" (apache#682) This reverts commit 035f4e4.
In #30, both @tgraves and I ran into the issue of building the assembly jar on a Red Hat system: the resulting jar does not load the python files properly, which were needed for running PySpark on YARN.
In the medium term, we should figure out what the issue is, since Red Hat is used quite commonly. For now, we should at the very least document it so people don't run into the same headaches that plagued us for days.