Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-4931][Yarn][Docs] Fix the format of running-on-yarn.md #3774

Closed
wants to merge 1 commit into from
Closed

[SPARK-4931][Yarn][Docs] Fix the format of running-on-yarn.md #3774

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/running-on-yarn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,18 +211,18 @@ settings and a restart of all node managers. Thus, this is not applicable to hos

To use a custom log4j configuration for the application master or executors, there are two options:

- upload a custom log4j.properties using spark-submit, by adding it to the "--files" list of files
- upload a custom `log4j.properties` using `spark-submit`, by adding it to the `--files` list of files
to be uploaded with the application.
- add "-Dlog4j.configuration=<location of configuration file>" to "spark.driver.extraJavaOptions"
(for the driver) or "spark.executor.extraJavaOptions" (for executors). Note that if using a file,
the "file:" protocol should be explicitly provided, and the file needs to exist locally on all
- add `-Dlog4j.configuration=<location of configuration file>` to `spark.driver.extraJavaOptions`
(for the driver) or `spark.executor.extraJavaOptions` (for executors). Note that if using a file,
the `file:` protocol should be explicitly provided, and the file needs to exist locally on all
the nodes.

Note that for the first option, both executors and the application master will share the same
log4j configuration, which may cause issues when they run on the same node (e.g. trying to write
to the same log file).

If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use "${spark.yarn.app.container.log.dir}" in your log4j.properties. For example, log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log. For streaming application, configuring RollingFileAppender and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.
If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use `spark.yarn.app.container.log.dir` in your log4j.properties. For example, `log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log`. For streaming application, configuring `RollingFileAppender` and setting file location to YARN's log directory will avoid disk overflow caused by large log file, and logs can be accessed using YARN's log utility.

# Important notes

Expand Down