Skip to content

R parallel Snippets

Jump to bottom
Tobias Kind edited this page Sep 27, 2015 · 22 revisions

This page features a loose collection of parallel R code that works (# run). However I am currently not using Rocker so R example code may fail at any momement with any update or package upgrade. All examples were deployed under R PRO 3.2.2 under Windows7 with clean installations.

Create 120 node doSNOW cluster

This example creates a doSNOW cluster with 120 nodes and plots a Gantt chart. IT will work on any computer. Under Windows each rscript client require 44 Mb memory (120*44=5,280 Mbyte), hence 6 GByte RAM is needed. Execute the following code and you shall be rewarded with the pretty picture below. For lesser memory machines try the example with 64 nodes. R source code is linked below.

### Create doSNOW cluster with 120 nodes and plot Gantt chart
### This works even on a single CPU computer
### Each rscript client requires 44 Mb memory, hence 6 GByte RAM needed.
### Example code from doSNOW package
### Tobias Kind (2015)

# Installation of the doSNOW parallel library with all dependencies
doInstall <- TRUE # Change to FALSE if you don't want packages installed.
toInstall <- c("doSNOW") 
if((doInstall) && (!is.element(toInstall, installed.packages()[,1])))
{
	cat("Please install required package. Select server:"); chooseCRANmirror();
	install.packages(toInstall, dependencies = c("Depends", "Imports")) 
}

# load doSnow library
library(doSNOW)

# create a socket cluster with 120 nodes (current max=128)
# See R source connection.c which defines max number of nodes
# define NCONNECTIONS 128 /* snow needs one per slave node */
cl <- makeCluster(120,type="SOCK")

# calculate some data points see plot(x)
x <- rnorm(1000000)

# time the snow cluster and plot Gantt chart (documentation)
# clusterCall calls a function fun with identical arguments on each node
tm <- snow.time(clusterCall(cl, function(x) for (i in 1:100) sum(x), x))

# print the node timings
print(tm)

#plot the  Gantt chart
plot(tm)

# stop the cluster
stopCluster(cl)

# remove the large x object and clean up memory
remove(x); gc()

### END

The link below is for quick access in PRO and Rstudio (https:// is not supported in R).

# Run the doSNOW 120 node cluster demo (copy/paste into R)
source(url("https://raw.githubusercontent.com/tobigithub/R-parallel/gh-pages/R/code-snippets/create-120node-doSNOW-cluster.R"))
# Total time: 10 seconds

dosnow-120-nodes-cluster

The figure above shows on the x-axis the time spent for calclations (in seconds) and on the y-axis the number of deployed nodes. On a higher clocked (4Ghz) Xeon or Core-i7 system the time maybe as low as six seconds. A very lovely figure indeed.

Source code:

Measure timing with 4/8/16/32/64/120 node doSNOW cluster

We can extend our measurements and test different cluster sizes. The test cluster here only has 16 physical cores and 32 threads and we can see on the picture below that using a cluster in this case does not really make sense, because the overhead is so large. Meaning, with larger cluster size the computational time also extends. You can checkout the source code below. Experiments would be to increase the matrix size (x <- rnorm(1000000)) to 1E9 to simulate several GByte to be loaded to each node or simply increasing the maxLoop number to 1E4 or 1E5. It is important to notice, that the only function used here is clusterCall(). This function calls a function fun with identical arguments on each node, hence no speed-up, just distribution in the current implementation.

dosnow-multiple-nodes

Source code: test-multiple-node-sizes-doSNOW-cluster.R

More to come...

Clone this wiki locally