Deploying

Deploying a StormCV topology is not different from deploying any other Storm topology and requires a running Storm cluster (Zookeeper, Nimbus and Workers) you can use. The first step below describes the steps to set up a cluster on your local machine. If you already have a cluster running you can skip this and continue with the tutorial describing how to set up the stormcv-deploy project, package it and deploy it on your Storm cluster.

Note: StormCV is based on Storm 0.9.2 and it is advised to deploy it on storm clusters with the same version (or 0.9.1 which was tested)

Step 1: set up a single node cluster

This section describes how to set up a single node cluster on your local machine. A more detailed description can be found in this tutorial. There are multiple resources on this topic so feel free to search yourself as well (be aware of older versions since they might not be valid anymore).

1. Download Zookeeper 3.4.6 at a convenient location

• open a terminal and start Zookeeper by executing zkServer start within the /bin directory

2. Download Storm 0.9.2-incubating from the Storm website and edit /conf/storm.yaml to specify your local machine to be both zookeeper and nimbus and add 4 'workers' like this:

storm.zookeeper.servers:
    - "localhost"
nimbus.host: "localhost"
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

3. open a new terminal and start Nimbus by executing storm nimbus in Storm's /bin directory
4. open a new terminal and start Supervisor by executing storm supervisor in Storm's /bin directory
5. Optional: open a new terminal and start Storm UI by executing storm ui in Storm's /bin directory. Go to localhost:8080 to view Storm's web console

At this point you should have at least three terminals open, each running one of the services started. In an operational setting it is advises to start all services under supervision but this is not needed for this tutorial.

Step 2: set up the stormcv-project

The repository contains a boilerplate Maven project called stormcv-deploy, follow these steps to set it up:

1. Open eclipse and install the Maven plugin:
   • Eclipse menu choose: Help > Install new software

• Choose work with: --All available sites--
• This should generate a list with available software which contains Maven integration for Eclipse
• Select this entry en click Next and install Maven

2. clone the project from the repository if you have not already done so git clone https://[email protected]/sensorstorm/stormcv.git
3. Open eclipse and import the project into the workspace (as a Maven project)
4. If Eclipse did not recognise it as a Maven project try to convert it:
   • right click on stromcv_deploy project

• choose Configure
• Convert to Maven project

Step 3: build the topology and package it as a JAR file.

The stormcv-deploy contains one class with a main method which contains the code to define and configure the topology. This code is almost the same as the code in the examples but should end with StormSubmitter as shown below:

// a single line does the actual topology deployment...
StormSubmitter.submitTopology("Your_topology_name", conf, builder.createTopology());

Build this project and package it using the following steps:

1. right-click on the project
2. choose Run as
3. choose Maven build
4. give package as build goal
5. and run

The screenshot below shows the build process in eclipse, the jar file (including dependencies!) can be found in the target directory.

This will create two jar files within the target directory of stormcv-deploy: one without and one with dependencies. The one including the dependencies can be deployed on Storm clusters!

Any additional resources used by the topology like model files can be placed in the src/main/resources directory which will be included in the jar file. Note, any opencv libraries build by yourself should also be put here so the platform can locate and load them! Also see this page for more information on topology building using maven.

Step 4: deploy the topology

The bin directory of your Storm download contains the storm executable needed to deploy topologies, also see the command line documentation). To deploy the jar file generated in step 3 you have to execute the line below, the actual paths depends on your local setup!.

storm jar path/to/jarfile/stormcv-deploy-0.0.1-SNAPSHOT-jar-with-dependencies.jar nl.tno.stormcv.deploy.DeploymentTopology

The topology is deployed on the nimbus node specified within the storm config file which is localhost if you executed step 1 above. When you open Storm's Web-UI and click your uploaded topology you will see something like the screenshot below. This screen contains a lot of information on the performance of your topology. It can be seen that the topology has been running for almost 3 minutes and the total time it takes from spout to streamer bolt is 1131 ms (including buffering etc).

By clicking the streamer you can find out on what node this bolt is running and you can access this node on port 8558 to view the webservice. Make sure there is no firewall blocking this port! The output should look like the screenshot below taken from a real cluster where this topology was executed on 13 cores (which is way to many as indicated by the low capacity scores).

Deploying on Amazon EC2

This section provides some starting points if you want to run topologies on Amazon’s EC2. First of all you need a login at aws.amazon.com and be familiar with Amazon AWS (have an account, ec2 credentials, keys etc). If you have no idea what is meant by this please sign up first and read this tutorial.

In order to run a topology on Amazon AWS (or any other cloud) you need to install and configure Storm on the EC2 instances you want to run everything on. In order to get this done you can use the storm-deploy-alternative developed by Kasper Madsen. This java program enables you to request EC2 instances of the type you want and install Storm 0.9.0.1 on them. Your EC2 credentials must be specified in credential.yaml.

The example configuration.yaml below configures a cluster of two g2.2xlarge servers running Ubuntu 12.04 within Amazons eu-west (Ireland) datacenter. The image type specifies what OS and configuration must be deployed on the servers which is important because you need to provide an OpenCV build for that specific OS. The linux OpenCV library provided with StormCV works on this specific image. Note that each region has its own AMI’s so if you want to deploy a cluster somewhere else you need to specify the right image id you can find here.

    storm:
    - provider "aws-ec2"
    - storm-version "0.9.0.1"
    - zk-version "3.4.5"	
    - image "eu-west-1/ami-7f08d908"		# Ubuntu 12.04 for g2.2xlarge 
    - region "eu-west-1"					# Region (Ireland)
    - g2.2xlarge {ZK, WORKER, MASTER, UI}	# Request service
    - g2.2xlarge {WORKER}					# Request service

Once the deploy script is done you have a running storm cluster which you can use to deploy your topologies on! Be patient when running the script since it might take 10 to 20 minutes to complete (depending on hardware). This script automatically configures the IP address of the Nimbus node so you can execute step 2 as described above directly when the script has finished.