Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH #5580

Closed
wants to merge 20 commits into from
Closed

[SPARK-6869][PySpark] Add pyspark archives path to PYTHONPATH #5580

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
f72987c
add archives path to PYTHONPATH
lianhuiwang Apr 16, 2015
9f31dac
update code and add comments
lianhuiwang Apr 19, 2015
31e8e06
update code style
lianhuiwang Apr 19, 2015
e0179be
add zip pyspark archives in build or sparksubmit
lianhuiwang Apr 26, 2015
0d2baf7
update import paths
lianhuiwang Apr 26, 2015
9396346
put zip to make-distribution.sh
lianhuiwang Apr 27, 2015
3b1e4c8
address tgravescs's comments
lianhuiwang Apr 27, 2015
5192cca
update import path
lianhuiwang Apr 27, 2015
f11f84a
zip pyspark archives
lianhuiwang Apr 29, 2015
e6b573b
address vanzin's comments
lianhuiwang Apr 29, 2015
4b8a3ed
use pyArchivesEnvOpt
lianhuiwang Apr 29, 2015
e7bd971
address vanzin's comments
lianhuiwang Apr 30, 2015
9d87c3f
update scala style
lianhuiwang Apr 30, 2015
20402cd
use ZipEntry
lianhuiwang Apr 30, 2015
150907b
Merge remote-tracking branch 'remotes/apache/master' into SPARK-6869
lianhuiwang May 3, 2015
f0b4ed8
Merge remote-tracking branch 'remotes/apache/master' into SPARK-6869
lianhuiwang May 3, 2015
008850a
Merge remote-tracking branch 'remotes/apache/master' into SPARK-6869
lianhuiwang May 3, 2015
1c8f664
Merge remote-tracking branch 'remotes/apache/master' into SPARK-6869
lianhuiwang May 3, 2015
c2ad0f9
Update Client.scala
lianhuiwang May 3, 2015
66ffa43
Update Client.scala
lianhuiwang May 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,14 @@ object SparkSubmit {
}
}

// In yarn mode for a python app, if PYSPARK_ARCHIVES_PATH is in the user environment
// add pyspark archives to files that can be distributed with the job
if (args.isPython && clusterManager == YARN){
sys.env.get("PYSPARK_ARCHIVES_PATH").map { archives =>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the user set this himself? What does he set it too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark administrator can set pyspark's zip of local or hdfs path to PYSPARK_ARCHIVES_PATH in spark-env.sh. and then user run python application as before.

args.files = mergeFileLists(args.files, Utils.resolveURIs(archives))
}
}

// If we're running a R app, set the main class to our specific R runner
if (args.isR && deployMode == CLIENT) {
if (args.primaryResource == SPARKR_SHELL) {
Expand Down
10 changes: 10 additions & 0 deletions yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,16 @@ private[spark] class Client(
distCacheMgr.setDistFilesEnv(env)
distCacheMgr.setDistArchivesEnv(env)

// If PYSPARK_ARCHIVES_PATH is in the user environment, set PYTHONPATH to be passed
// on to the ApplicationMaster and the executors.
sys.env.get("PYSPARK_ARCHIVES_PATH").map { archives =>
// archives will be distributed to each machine's working directory, so strip the
// path prefix
val pythonPath = archives.split(",").map(p => (new Path(p)).getName).mkString(":")
env("PYTHONPATH") = pythonPath
sparkConf.setExecutorEnv("PYTHONPATH", pythonPath)
}

// Pick up any environment variables for the AM provided through spark.yarn.appMasterEnv.*
val amEnvPrefix = "spark.yarn.appMasterEnv."
sparkConf.getAll
Expand Down