[SPARK-1755] Respect SparkSubmit --name on YARN #699
Conversation
|
Merged build triggered. |
|
Merged build started. |
|
Merged build triggered. |
|
Merged build started. |
|
LGTM. My only concern (which I mentioned in pr #539) is that an app calling SparkConf.setAppName() will end up overriding this command line option, right? That seems a little counter-intuitive to me (I'd expect the command line to take precedence), but not a big deal. |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14817/ |
|
I actually think an app name explicitly set by the application itself should take precedence. It's a little strange to write an application, specify a name, and possibly have it overridden somewhere else. Otherwise we could potentially give |
|
Merged build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14818/ |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
I've merged this thanks. |
Right now, SparkSubmit ignores the `--name` flag for both yarn-client and yarn-cluster. This is a bug. In client mode, SparkSubmit treats `--name` as a [cluster config](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170) and does not propagate this to SparkContext. In cluster mode, SparkSubmit passes this flag to `org.apache.spark.deploy.yarn.Client`, which only uses it for the [YARN ResourceManager](https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80), but does not propagate this to SparkContext. This PR ensures that `spark.app.name` is always set if SparkSubmit receives the `--name` flag, which is what the usage promises. This makes it possible for applications to start a SparkContext with an empty conf `val sc = new SparkContext(new SparkConf)`, and inherit the app name from SparkSubmit. Tested both modes on a YARN cluster. Author: Andrew Or <[email protected]> Closes #699 from andrewor14/yarn-app-name and squashes the following commits: 98f6a79 [Andrew Or] Fix tests dea932f [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-app-name c86d9ca [Andrew Or] Respect SparkSubmit --name on YARN (cherry picked from commit 8b78412) Signed-off-by: Patrick Wendell <[email protected]>
Right now, SparkSubmit ignores the `--name` flag for both yarn-client and yarn-cluster. This is a bug. In client mode, SparkSubmit treats `--name` as a [cluster config](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170) and does not propagate this to SparkContext. In cluster mode, SparkSubmit passes this flag to `org.apache.spark.deploy.yarn.Client`, which only uses it for the [YARN ResourceManager](https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80), but does not propagate this to SparkContext. This PR ensures that `spark.app.name` is always set if SparkSubmit receives the `--name` flag, which is what the usage promises. This makes it possible for applications to start a SparkContext with an empty conf `val sc = new SparkContext(new SparkConf)`, and inherit the app name from SparkSubmit. Tested both modes on a YARN cluster. Author: Andrew Or <[email protected]> Closes apache#699 from andrewor14/yarn-app-name and squashes the following commits: 98f6a79 [Andrew Or] Fix tests dea932f [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-app-name c86d9ca [Andrew Or] Respect SparkSubmit --name on YARN
…pache#701) * minor fix ut * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 (apache#691) * AL-8998 Fix resizing for ArrayBuffer in scala-2.12 * KE-42864 Fix vulnerability, upgrade tomcat-embed-core.version version to 9.0.81 (apache#699) --------- Co-authored-by: jiawei.li <[email protected]> Co-authored-by: Zhimin Wu <[email protected]>
Right now, SparkSubmit ignores the
--nameflag for both yarn-client and yarn-cluster. This is a bug.In client mode, SparkSubmit treats
--nameas a cluster config and does not propagate this to SparkContext.In cluster mode, SparkSubmit passes this flag to
org.apache.spark.deploy.yarn.Client, which only uses it for the YARN ResourceManager, but does not propagate this to SparkContext.This PR ensures that
spark.app.nameis always set if SparkSubmit receives the--nameflag, which is what the usage promises. This makes it possible for applications to start a SparkContext with an empty confval sc = new SparkContext(new SparkConf), and inherit the app name from SparkSubmit.Tested both modes on a YARN cluster.