Skip to content

Latest commit

 

History

History
61 lines (53 loc) · 2.43 KB

README.md

File metadata and controls

61 lines (53 loc) · 2.43 KB

Introduction

Spark-on-HPC dynamically provisions Apache Spark clusters and run spark jobs on an HPC under its traditional resource manager. Currently, PBS or Torque on a linux cluster are supported.

Features

  • Run under PBS resource limit, i.e. number of nodes, number of cores, memory and walltime
  • Multiple spark jobs (master port is selected randomly for each job) for any user
  • Only master and workers of the same job are allowed to connect together by a shared secret.

Requirements

  • Linux (should work with most distributions)
  • Apache Spark 1.3.0+

Installation

  • Download and unpack Spark package into SPARK_HOME directory
  • Download and unzip the Spark-on-HPC package. Change directory to SPARK_ON_HPC root directory
#cd $SPARK_ON_HPC
  • Copy scripts into $SPARK_HOME/sbin
#cp pbs/spark-sbin/* $SPARK_HOME/sbin

Usage

Root permission is NOT required.

  • Once installed, create a job directory.
#cd $HOME
#mkdir test
#cd test
  • Copy an example job script inside the package. There are two examples in the package. One is for single node script. The other is for multiple node script.
#cp $SPARK_ON_HPC/examples/test_spark_multi/spark_multi.sh test_spark_job.sh
  • Make change to the script. Usually, the directives, shell variables and spark-submit arguments are changed. Set directives. For a PBS example, request 5 nodes (1 master + 4 workers), each with 1gb memory allocated, and queue "test".
#PBS -l nodes=5
#PBS -l pmem=1gb
#PBS -q test
  • In job script, set SPARK_HOME to where the spark package is installed, and SPARK_JOB_DIR to the directory where the configuration and log files will be created. Note that the PBS_O_WORKDIR is the location where qsub command is executed.
export SPARK_HOME=$HOME/spark-1.4.1-bin-1.2.1
export SPARK_JOB_DIR=$PBS_O_WORKDIR
  • In job script, change the spark-submit arguments. For example, running JavaSparkPi 10 tasks
SPARK_HOME/bin/spark-submit --master $SPARK_URL --class org.apache.spark.examples.JavaSparkPi $SPARK_HOME/lib/spark-examples-1.4.1-hadoop1.2.1.jar 10 > $PBS_O_WORKDIR/pi.txt
  • Submit a job
#qsub test_spark_job.sh

The directory conf, logs, and work will be created in SPARK_JOB_DIR during the execution of spark. Examine them if necessary in addition to the normal job stdout and stderr files.

How to run spark-on-hpc manually for testing purpose

Set environment variables SPARK_JOB_DIR, SPARK_HOME, and PBS_NODEFILE.