SPARK-1652: Set driver memory correctly in spark-submit.

The previous check didn't account for the fact that the default deploy mode is "client" unless otherwise specified. Also, this sets the more narrowly defined SPARK_DRIVER_MEM instead of setting SPARK_MEM.
jegonzal · May 11, 2014 · f508146 · f508146
1 parent 70bcdef
commit f508146
Showing 1 changed file with 2 additions and 2 deletions.
diff --git a/bin/spark-submit b/bin/spark-submit
@@ -35,8 +35,8 @@ while (($#)); do
   shift
 done
 
-if [ ! -z $DRIVER_MEMORY ] && [ ! -z $DEPLOY_MODE ] && [ $DEPLOY_MODE = "client" ]; then
-  export SPARK_MEM=$DRIVER_MEMORY
+if [ ! -z $DRIVER_MEMORY ] && [ ! "$DEPLOY_MODE" == "cluster" ]; then
+  export SPARK_DRIVER_MEMORY=$DRIVER_MEMORY
 fi
 
 $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit "${ORIG_ARGS[@]}"