From 88fd0294ec1562827347934dabb9d801e07e54ef Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Mon, 23 Aug 2021 10:33:53 -0700 Subject: [PATCH] Docs: Cori PICMI Instructions (#2219) * Docs: Cori PICMI Instructions Too tricky to get right to just cross-link - add documentation how to build & run PICMI interfaces on Cori. * Link: Jupyter Instructions --- Docs/source/install/hpc/cori.rst | 118 ++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 24 deletions(-) diff --git a/Docs/source/install/hpc/cori.rst b/Docs/source/install/hpc/cori.rst index 6b9d9d18a3c..014a446dedf 100644 --- a/Docs/source/install/hpc/cori.rst +++ b/Docs/source/install/hpc/cori.rst @@ -9,9 +9,10 @@ If you are new to this system, please see the following resources: * `GPU nodes `__ -* `Cori user guide `_ -* Batch system: `Slurm `_ -* `Production directories `_: +* `Cori user guide `__ +* Batch system: `Slurm `__ +* `Jupyter service `__ +* `Production directories `__: * ``$SCRATCH``: per-user production directory (20TB) * ``/global/cscratch1/sd/m3239``: shared production directory for users in the project ``m3239`` (50TB) @@ -37,10 +38,16 @@ We use the following modules and environments on the system (``$HOME/knl_warpx.p module swap PrgEnv-intel PrgEnv-gnu module load cmake/3.20.5 module load cray-hdf5-parallel/1.10.5.2 - module load cray-fftw + module load cray-fftw/3.3.8.4 module load cray-python/3.7.3.2 - export CMAKE_PREFIX_PATH=$PWD/adios2-2.7.1-knl-install:$CMAKE_PREFIX_PATH + export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH + export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-knl-install:$CMAKE_PREFIX_PATH + + if [ -d "$HOME/sw/venvs/knl_warpx" ] + then + source $HOME/sw/venvs/knl_warpx/bin/activate + fi export CXXFLAGS="-march=knl" export CFLAGS="-march=knl" @@ -51,9 +58,23 @@ And install ADIOS2: source $HOME/knl_warpx.profile - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git adios2 - cmake -S adios2 -B adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=adios2-2.7.1-knl-install - cmake --build adios2-build --target install --parallel 16 + git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 + cmake -S src/adios2 -B src/adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/adios2-2.7.1-knl-install + cmake --build src/adios2-build --target install --parallel 16 + +For PICMI and Python workflows, also install a virtual environment: + +.. code-block:: bash + + # establish Python dependencies + python3 -m pip install --user --upgrade pip + python3 -m pip install --user virtualenv + + python3 -m venv $HOME/sw/venvs/knl_warpx + source $HOME/sw/venvs/knl_warpx/bin/activate + + python3 -m pip install --upgrade pip + MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py Haswell ^^^^^^^ @@ -62,12 +83,19 @@ We use the following modules and environments on the system (``$HOME/haswell_war .. code-block:: bash + module swap PrgEnv-intel PrgEnv-gnu module load cmake/3.20.5 module load cray-hdf5-parallel/1.10.5.2 - module load cray-fftw + module load cray-fftw/3.3.8.4 module load cray-python/3.7.3.2 - export CMAKE_PREFIX_PATH=$PWD/adios2-2.7.1-haswell-install:$CMAKE_PREFIX_PATH + export PKG_CONFIG_PATH=$FFTW_DIR/pkgconfig:$PKG_CONFIG_PATH + export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-haswell-install:$CMAKE_PREFIX_PATH + + if [ -d "$HOME/sw/venvs/haswell_warpx" ] + then + source $HOME/sw/venvs/haswell_warpx/bin/activate + fi And install ADIOS2: @@ -75,9 +103,23 @@ And install ADIOS2: source $HOME/haswell_warpx.profile - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git adios2 - cmake -S adios2 -B adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=adios2-2.7.1-haswell-install - cmake --build adios2-build --target install --parallel 16 + git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 + cmake -S src/adios2 -B src/adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/adios2-2.7.1-haswell-install + cmake --build src/adios2-build --target install --parallel 16 + +For PICMI and Python workflows, also install a virtual environment: + +.. code-block:: bash + + # establish Python dependencies + python3 -m pip install --user --upgrade pip + python3 -m pip install --user virtualenv + + python3 -m venv $HOME/sw/venvs/haswell_warpx + source $HOME/sw/venvs/haswell_warpx/bin/activate + + python3 -m pip install --upgrade pip + MPICC="cc -shared" python3 -m pip install -U --no-cache-dir -v mpi4py GPU (V100) ^^^^^^^^^^ @@ -96,7 +138,12 @@ We use the following modules and environments on the system (``$HOME/gpu_warpx.p module load gcc/8.3.0 cuda/11.4.0 cmake/3.20.5 module load openmpi - export CMAKE_PREFIX_PATH=$PWD/adios2-2.7.1-gpu-install:$CMAKE_PREFIX_PATH + export CMAKE_PREFIX_PATH=$HOME/sw/adios2-2.7.1-gpu-install:$CMAKE_PREFIX_PATH + + if [ -d "$HOME/sw/venvs/gpu_warpx" ] + then + source $HOME/sw/venvs/gpu_warpx/bin/activate + fi # compiler environment hints export CC=$(which gcc) @@ -120,10 +167,23 @@ And install ADIOS2: source $HOME/gpu_warpx.profile - git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git adios2 - cmake -S adios2 -B adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=adios2-2.7.1-gpu-install - cmake --build adios2-build --target install --parallel 16 + git clone -b v2.7.1 https://github.com/ornladios/ADIOS2.git src/adios2 + cmake -S src/adios2 -B src/adios2-build -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DCMAKE_INSTALL_PREFIX=$HOME/sw/adios2-2.7.1-gpu-install + cmake --build src/adios2-build --target install --parallel 16 + +For PICMI and Python workflows, also install a virtual environment: + +.. code-block:: bash + + # establish Python dependencies + python3 -m pip install --user --upgrade pip + python3 -m pip install --user virtualenv + python3 -m venv $HOME/sw/venvs/gpu_warpx + source $HOME/sw/venvs/gpu_warpx/bin/activate + + python3 -m pip install --upgrade pip + python3 -m pip install -U --no-cache-dir -v mpi4py Building WarpX -------------- @@ -142,6 +202,15 @@ If you want to run on either of the three partitions of Cori, open a new termina # GPU: #source $HOME/gpu_warpx.profile +.. warning:: + + Consider that all three Cori partitions are *incompatible*. + + Do not *source* multiple ``...warpx.profile`` files in the same terminal session. + Open a new terminal and log into Cori again, if you want to switch the targeted Cori partition. + + If you re-submit an already compiled simulation that you ran on another day or in another session, *make sure to source* the corresponding ``...warpx.profile`` again after login! + Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following commands to compile: .. code-block:: bash @@ -153,16 +222,15 @@ Then, ``cd`` into the directory ``$HOME/src/warpx`` and use the following comman cmake -S . -B build -DWarpX_OPENPMD=ON -DWarpX_DIMS=3 cmake --build build -j 16 -The general :ref:`cmake compile-time options and instructions for Python (PICMI) bindings ` apply as usual. +The general :ref:`cmake compile-time options and instructions for Python (PICMI) bindings ` apply as usual: -.. warning:: - - Consider that all three Cori partitions are *incompatible*. +.. code-block:: bash - Do not *source* multiple ``...warpx.profile`` files in the same terminal session. - Open a new terminal and log into Cori again, if you want to switch the targeted Cori partition. + # PICMI build + cd $HOME/src/warpx - If you re-submit an already compiled simulation that you ran on another day or in another session, *make sure to source* the corresponding ``...warpx.profile`` again after login! + # compile parallel PICMI interfaces with openPMD support and 3D, 2D and RZ + WarpX_MPI=ON WarpX_OPENPMD=ON BUILD_PARALLEL=16 python3 -m pip install --force-reinstall -v . .. _running-cpp-cori: @@ -178,6 +246,8 @@ The batch script below can be used to run a WarpX simulation on 2 KNL nodes on the supercomputer Cori at NERSC. Replace descriptions between chevrons ``<>`` by relevant values, for instance ```` could be ``laserWakefield``. +For PICMI Python runs, the ```` has to read ``python3`` and the ```` is the path to your PICMI input script. + .. literalinclude:: ../../../../Tools/BatchScripts/batch_cori.sh :language: bash