[SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. #701
Conversation
|
Merged build triggered. |
|
Merged build started. |
| @@ -43,18 +43,19 @@ Unlike in Spark standalone and Mesos mode, in which the master's address is spec | |||
|
|
|||
| To launch a Spark application in yarn-cluster mode: | |||
|
|
|||
| ./bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] <app jar> [app options] | |||
| ./bin/spark-submit --class path.to.your.Class --master yarn-cluster --deploy-mode cluster [options] <app jar> [app options] | |||
There was a problem hiding this comment.
If using --deploy-mode cluster, then --master should just be "yarn".
There was a problem hiding this comment.
Ah, I didn't realize master=yarn-cluster also sets deployMode
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
| @@ -82,6 +82,12 @@ will print out the contents of all log files from all containers from the given | |||
|
|
|||
| When log aggregation isn't turned on, logs are retained locally on each machine under YARN_APP_LOGS_DIR, which is usually configured to /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version and installation. Viewing logs for a container requires going to the host that contains them and looking in this directory. Subdirectories organize log files by application ID and container ID. | |||
|
|
|||
| To review per container launch environment, increase yarn.nodemanager.delete.debug-delay-sec to a | |||
There was a problem hiding this comment.
Nit: per-container should be hyphenated
There was a problem hiding this comment.
This isn't available to all users. This only applies if you are running your own cluster and have control over nodemanager settings. I believe it also requires a nodemanager restart. on a hosted cluster you won't be able to change this so I think we should add something about that.
There was a problem hiding this comment.
Good point, will do.
|
Added a minor nit. Otherwise, this looks good to me. |
andrewor14
changed the title
|
Merged build triggered. |
|
Merged build started. |
|
thanks for the update, the changes look good to me. I did notice one other thing in the yarn docs and that is that we don't say what parameters spark-shell takes. I know it used to list the environment variables to use to configure but now it doesn't say how to configure and I don't see any other docs that say how to do that. I actually went to look at the code to see how it is now done. So perhaps I'll file a separate jira for that unless you know of another pr perhaps handling that? |
|
AFAIK the latest spark-shell passes its arguments directly to spark-submit, so in that sense spark-shell inherits whatever parameters spark-submit takes in. scala-programming-guide discusses this very briefly. Not sure if this was what you were expecting? Though we should probably document that spark-shell parameters are basically spark-submit parameters, and add a link between the two. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
I see where the programming guide mentions a couple of options but I think it would be good to explicitly say it since people moving from 0.9 to 1.0 might not know that. |
andrewor14
changed the title
andrewor14
changed the title
|
Merged build triggered. |
This commit removes the section on using org.apache.spark.deploy.Client to launch an application. This is subsumed by the Spark submit section immediately preceding it. This commit also clarifies how we set the Spark configuration properties in the 1.0 world. Previously it was pretty unclear, and the necessary details were in the wrong page (cluster-overview.html) instead of where it is supposed to be (configuration.html).
|
Merged build triggered. |
|
Merged build started. |
andrewor14
changed the title
|
Merged build triggered. |
|
Merged build started. |
andrewor14
changed the title
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14921/ |
…N, standalone etc. YARN - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that. - The default YARN build guide should be in maven, not sbt. - This PR also adds a paragraph on steps to debug a YARN application. Standalone - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`. - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made. In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there. Author: Andrew Or <[email protected]> Closes #701 from andrewor14/yarn-docs and squashes the following commits: e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814) 25cfe7b [Andrew Or] Merge in the warning from SPARK-1753 a8c39c5 [Andrew Or] Minor changes 336bbd9 [Andrew Or] Tabs -> spaces 4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html 3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions 5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc. c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 381fe32 [Andrew Or] Update docs for standalone mode 757c184 [Andrew Or] Add a note about the requirements for the debugging trick f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 924f04c [Andrew Or] Revert addition of --deploy-mode d5fe17b [Andrew Or] Update the YARN docs (cherry picked from commit 2ffd1ea) Signed-off-by: Patrick Wendell <[email protected]>
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
…N, standalone etc. YARN - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that. - The default YARN build guide should be in maven, not sbt. - This PR also adds a paragraph on steps to debug a YARN application. Standalone - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`. - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made. In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there. Author: Andrew Or <[email protected]> Closes apache#701 from andrewor14/yarn-docs and squashes the following commits: e2c2312 [Andrew Or] Merge in changes in apache#752 (SPARK-1814) 25cfe7b [Andrew Or] Merge in the warning from SPARK-1753 a8c39c5 [Andrew Or] Minor changes 336bbd9 [Andrew Or] Tabs -> spaces 4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html 3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions 5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc. c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 381fe32 [Andrew Or] Update docs for standalone mode 757c184 [Andrew Or] Add a note about the requirements for the debugging trick f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs 924f04c [Andrew Or] Revert addition of --deploy-mode d5fe17b [Andrew Or] Update the YARN docs
…tdout (apache#701) Co-authored-by: helenyugithub <[email protected]>
…pache#701) * minor fix ut * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 (apache#691) * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 * KE-42864 Fix vulnerability, upgrade tomcat-embed-core.version version to 9.0.81 (apache#699) --------- Co-authored-by: jiawei.li <[email protected]> Co-authored-by: Zhimin Wu <[email protected]>
YARN
Standalone
org.apache.spark.deploy.Client.In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.