diff --git a/docs/openstack-integration.md b/docs/openstack-integration.md new file mode 100644 index 0000000000000..42cd3067edf80 --- /dev/null +++ b/docs/openstack-integration.md @@ -0,0 +1,83 @@ +--- +layout: global +title: Accessing Openstack Swift storage from Spark +--- + +# Accessing Openstack Swift storage from Spark + +Spark's file interface allows it to process data in Openstack Swift using the same URI formats that are supported for Hadoop. You can specify a path in Swift as input through a URI of the form `swift:///path`. You will also need to set your Swift security credentials, through `SparkContext.hadoopConfiguration`. + +#Configuring Hadoop to use Openstack Swift +Openstack Swift driver was merged in Hadoop verion 2.3.0 ([Swift driver](https://issues.apache.org/jira/browse/HADOOP-8545)) Users that wish to use previous Hadoop versions will need to configure Swift driver manually. +

Hadoop 2.3.0 and above.

+An Openstack Swift driver was merged into Haddop 2.3.0 . Current Hadoop driver requieres Swift to use Keystone authentication. There are additional efforts to support temp auth for Hadoop [Hadoop-10420](https://issues.apache.org/jira/browse/HADOOP-10420). +To configure Hadoop to work with Swift one need to modify core-sites.xml of Hadoop and setup Swift FS. + + + + fs.swift.impl + org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem + + + + +

Configuring Spark - stand alone cluster

+You need to configure the compute-classpath.sh and add Hadoop classpath for + + + CLASSPATH = /share/hadoop/common/lib/* + CLASSPATH = /share/hadoop/hdfs/* + CLASSPATH = /share/hadoop/tools/lib/* + CLASSPATH = /share/hadoop/hdfs/lib/* + CLASSPATH = /share/hadoop/mapreduce/* + CLASSPATH = /share/hadoop/mapreduce/lib/* + CLASSPATH = /share/hadoop/yarn/* + CLASSPATH = /share/hadoop/yarn/lib/* + +Additional parameters has to be provided to the Hadoop from Spark. Swift driver of Hadoop uses those parameters to perform authentication in Keystone needed to access Swift. +List of mandatory parameters is : `fs.swift.service..auth.url`, `fs.swift.service..auth.endpoint.prefix`, `fs.swift.service..tenant`, `fs.swift.service..username`, +`fs.swift.service..password`, `fs.swift.service..http.port`, `fs.swift.service..http.port`, `fs.swift.service..public`. +Create core-sites.xml and place it under /spark/conf directory. Configure core-sites.xml with general Keystone parameters, for example + + + + fs.swift.service..auth.url + http://127.0.0.1:5000/v2.0/tokens + + + fs.swift.service..auth.endpoint.prefix + endpoints + + fs.swift.service..http.port + 8080 + + + fs.swift.service..region + RegionOne + + + fs.swift.service..public + true + + +We left with `fs.swift.service..tenant`, `fs.swift.service..username`, `fs.swift.service..password`. The best way is to provide them to SparkContext in run time, which seems to be impossible yet. +Another approach is to change Hadoop Swift FS driver to provide them via system environment variables. For now we provide them via core-sites.xml + + + fs.swift.service..tenant + test + + + fs.swift.service..username + tester + + + fs.swift.service..password + testing + + +

Usage

+Assume you have a Swift container `logs` with an object `data.log`. You can use `swift://` scheme to access objects from Swift. + + val sfdata = sc.textFile("swift://logs./data.log") +