[SPARK-17964][SparkR] Enable SparkR with Mesos client mode and cluster mode #15700
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mgummelt
approved these changes
Nov 1, 2016
| @@ -84,7 +84,7 @@ private[spark] object RUtils { | |||
| } | |||
| } else { | |||
| // Otherwise, assume the package is local | |||
| // TODO: support this for Mesos | |||
| // For Mesos, the path is also under SPARK_HOME. | |||
There was a problem hiding this comment.
This comment seems unnecessary now. Maybe remove it?
There was a problem hiding this comment.
I have removed this comment.
|
ok to test |
|
@srowen LGTM |
|
This might not handle packages the best it can on mesos but that could be a follow-up |
|
@felixcheung There are separate issues for packages that are not included in this PR: Also, I could not find any documentation about packages. It would be helpful to fill in some information about requirements or testing instructions in the links above. |
|
Test build #3407 has finished for PR 15700 at commit
|
|
retest this please |
|
Test build #3413 has finished for PR 15700 at commit
|
|
retest this please |
|
Jenkins add to whitelist |
|
Jenkins retest this please |
|
Test build #68200 has finished for PR 15700 at commit
|
|
Test build #68202 has finished for PR 15700 at commit
|
|
Merged to master/2.1 |
asfgit
pushed a commit
that referenced
this pull request
Nov 5, 2016
…r mode ## What changes were proposed in this pull request? Enabled SparkR with Mesos client mode and cluster mode. Just a few changes were required to get this working on Mesos: (1) removed the SparkR on Mesos error checks and (2) do not require "--class" to be specified for R apps. The logic to check spark.mesos.executor.home was already in there. sun-rui ## How was this patch tested? 1. SparkSubmitSuite 2. On local mesos cluster (on laptop): ran SparkR shell, spark-submit client mode, and spark-submit cluster mode, with the "examples/src/main/R/dataframe.R" example application. 3. On multi-node mesos cluster: ran SparkR shell, spark-submit client mode, and spark-submit cluster mode, with the "examples/src/main/R/dataframe.R" example application. I tested with the following --conf values set: spark.mesos.executor.docker.image and spark.mesos.executor.home This contribution is my original work and I license the work to the project under the project's open source license. Author: Susan X. Huynh <[email protected]> Closes #15700 from susanxhuynh/susan-r-branch. (cherry picked from commit 9a87c31) Signed-off-by: Sean Owen <[email protected]>
uzadude
pushed a commit
to uzadude/spark
that referenced
this pull request
Jan 27, 2017
…r mode ## What changes were proposed in this pull request? Enabled SparkR with Mesos client mode and cluster mode. Just a few changes were required to get this working on Mesos: (1) removed the SparkR on Mesos error checks and (2) do not require "--class" to be specified for R apps. The logic to check spark.mesos.executor.home was already in there. sun-rui ## How was this patch tested? 1. SparkSubmitSuite 2. On local mesos cluster (on laptop): ran SparkR shell, spark-submit client mode, and spark-submit cluster mode, with the "examples/src/main/R/dataframe.R" example application. 3. On multi-node mesos cluster: ran SparkR shell, spark-submit client mode, and spark-submit cluster mode, with the "examples/src/main/R/dataframe.R" example application. I tested with the following --conf values set: spark.mesos.executor.docker.image and spark.mesos.executor.home This contribution is my original work and I license the work to the project under the project's open source license. Author: Susan X. Huynh <[email protected]> Closes apache#15700 from susanxhuynh/susan-r-branch.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Enabled SparkR with Mesos client mode and cluster mode. Just a few changes were required to get this working on Mesos: (1) removed the SparkR on Mesos error checks and (2) do not require "--class" to be specified for R apps. The logic to check spark.mesos.executor.home was already in there.
@sun-rui
How was this patch tested?
This contribution is my original work and I license the work to the project under the project's open source license.