HCPAR (HAIL-CAESAR PARALLEL)

This is the HPC development version of HAIL-CAESAR arrived at by porting the code at https://github.com/dvalters/hail-caesar to use LibGeoDecomp for MPI-based multi-node parallelism.

Requirements

To build HCPAR you will need the following installed:

• g++ (more recent than version 6)

• an MPI library (e.g. MPICH, OpenMPI, ...)

• Boost C++ libraries

• PnetCDF (tested with version 1.12.2): https://parallel-netcdf.github.io/wiki/Download.html

• LibGeoDecomp with support for PnetCDF (see below)

LibGeoDecomp with PnetCDF support

PnetCDF needs to already be installed.

Download LibGeoDecomp with PnetCDF support (pnetcdf branch) from https://github.com/aproeme/libgeodecomp

To build a basic MPI version of LibGeoDecomp with minimal other features other than PnetCDF (e.g. no CUDA support), try the following (which assumes an installation into ~/libgeodecomp/with_pnetcdf):

mkdir build
cd build

cmake -Wno-dev \ 
  -DWITH_PNETCDF=true \
  -DPnetCDF_PATH=/path/to/your/pnetcdf/installation \
  -DWITH_SILO=false \
  -DWITH_VISIT=false \
  -DWITH_SCOTCH=false \
  -DWITH_OPENCL=false \
  -DWITH_CUDA=false \
  -DWITH_HPX=false \
  -DWITH_OPENCV=false \
  -DWITH_FORTRAN=false  \
  -DWITH_QT5=false \
  -DCMAKE_INSTALL_PREFIX=~/libgeodecomp/with_pnetcdf \
  -DCMAKE_CXX_COMPILER=mpic++ \
  -DCMAKE_C_COMPILER=mpicc \
  .. 

make
make install

Build

Edit make/make.inc and specify the install locations of the MPI library, Boost, and LibGeoDecomp with PnetCDF support.

Run make to build the hcpar executable in bin

Optionally, run make debug to build the hcpar executable in bin/debug/. The debug build uses -g -Og instead of the default -O3 to facilitate debugging.

Note: make clean works

Run

Run hcpar using the parallel application launcher associated with your MPI library (mpirun, mpiexec, aprun, etc.) as follows, e.g. to run on 256 cores:

mpirun -n 256 hcpar filename.params

For example parameter (.params) and other input files, see the tests directory

Develop

If you develop the code and modify, remove or add any Cell variables (grid quantities) in cell.hpp you must regenerate the MPI typemaps typemaps.cpp and typemaps.h prior to building hcpar (either the default target or debug executable). This can be done by running make typemaps, which requires doxygen and ruby.