Skip to content

Commit

Permalink
review comments
Browse files Browse the repository at this point in the history
  • Loading branch information
tgravescs committed Jun 23, 2014
1 parent 0ac8501 commit ddbcd16
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 3 deletions.
2 changes: 1 addition & 1 deletion docs/running-on-yarn.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
<td><code>spark.yarn.access.namenodes</code></td>
<td>(none)</td>
<td>
A list of secure HDFS namenodes your spark application is going to access. For example, spark.yarn.access.namenodes=hdfs://nn1.com:8032,hdfs://nn2.com:8032. Spark acquires security Tokens for each of the namenodes so that the spark application can access those remote HDFS clusters.
A list of secure HDFS namenodes your Spark application is going to access. For example, `spark.yarn.access.namenodes=hdfs://nn1.com:8032,hdfs://nn2.com:8032`. The Spark application must have acess to the namenodes listed and Kerberos must be properly configured to be able to access them (either in the same realm or in a trusted realm). Spark acquires security tokens for each of the namenodes so that the Spark application can access those remote HDFS clusters.
</td>
</tr>
</table>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -601,7 +601,9 @@ object ClientBase extends Logging {
YarnSparkHadoopUtil.addToEnvironment(env, Environment.CLASSPATH.name, path,
File.pathSeparator)

// get the list of namenodes the user may access
/**
* Get the list of namenodes the user may access.
*/
private[yarn] def getNameNodesToAccess(sparkConf: SparkConf): Set[Path] = {
sparkConf.get("spark.yarn.access.namenodes", "").split(",").map(_.trim()).filter(!_.isEmpty)
.map(new Path(_)).toSet
Expand All @@ -618,7 +620,9 @@ object ClientBase extends Logging {
delegTokenRenewer
}

// obtains tokens for the namenodes passed in and adds them to the credentials
/**
* Obtains tokens for the namenodes passed in and adds them to the credentials.
*/
private[yarn] def obtainTokensForNamenodes(paths: Set[Path], conf: Configuration,
creds: Credentials) {
if (UserGroupInformation.isSecurityEnabled()) {
Expand Down

0 comments on commit ddbcd16

Please sign in to comment.