Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding an MPI Hello World example in documentation on how to run on top of EESSI without EB #369

Merged
merged 6 commits into from
Jan 9, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 86 additions & 1 deletion docs/using_eessi/building_on_eessi.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,13 +143,98 @@ shell before building. To do this:
* Set manually `LD_RUN_PATH` to resolve libraries at runtime. `LIBRARY_PATH` should contain all the paths we need, and we also need to include the path to
`libstdc++` from our GCC installation to avoid picking up the one from the host:
```sh
export LD_RUN_PATH=$LIBRARY_PATH:$EBROOTGCCcore/lib64
export LD_RUN_PATH=$LIBRARY_PATH:$EBROOTGCCCORE/lib64
```
* Compile and make sure the library resolution points to the EESSI stack. For this, `ldd` from compatibility layer and **not** `/usr/bin/ldd` should be used
when checking the binary.

* Run!

To exemplify this, take the classic MPI Hello World example code:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thought to put it apart from the step-by-step explanation as it might be too verbose to put inside bullet points, any suggestion on how to do it better is appreciated


```
/*The Parallel Hello World Program*/
#include <stdio.h>
#include <mpi.h>

int main(int argc, char **argv)
{
int node;

MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

printf("Hello World from MPI rank %d\n", rank);

MPI_Finalize();
}

```

As described in the steps above, prepare the environment and load the required dependencies. For this case, we will use `gompi/2023b` as the toolchain to compile it.

```
# Starting the environment
$ source /cvmfs/software.eessi.io/versions/2023.06/init/bash

# Loading the toolchain
{EESSI 2023.06} $ module load gompi/2023b
```

Now, set the `LD_RUN_PATH` environment variable for all the libraries to point to the runtime libraries, then compile the code.

```
# Setting LD_RUN_PATH
{EESSI 2023.06}$ export LD_RUN_PATH=$LIBRARY_PATH:$EBROOTGCCCORE/lib64

# Compile the code manually
{EESSI 2023.06} $ mpicc -o HelloWorld mpi.c
```

This is the moment to check if the compiler picked all the libraries from the software and compatibility layer, not the host.

Look at the difference on the library solving when using the compatibility layer ldd from the host one:

```sh

# ldd from the compatibility layer, notice how all libraries are resolved from the software layer -> "/cvmfs/software.eessi.io/versions/2023.06/software", the libc and the interpreter point to the compatibility layer, so all good to go!

{EESSI 2023.06} $ ldd HelloWorld
linux-vdso.so.1 (0x00007ffce03af000)
libmpi.so.40 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/OpenMPI/4.1.6-GCC-13.2.0/lib/libmpi.so.40 (0x00007fadd9e84000)
libc.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/libc.so.6 (0x00007fadd9ca8000)
[...]
libevent_pthreads-2.1.so.7 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/libevent/2.1.12-GCCcore-13.2.0/lib64/libevent_pthreads-2.1.so.7 (0x00007fadd98f0000)
libm.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib/../lib64/libm.so.6 (0x00007fadd9810000)
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x00007fadd9fab000)

```

```sh

# ldd from the host, even though the libraries point to the software layer, now the linker ld-linux-x86-64.so.2 from the compat layer directly points to "/lib64/ld-linux-x86-64.so.2" from the host to do the resolving, resulting in the GLIBC mismatch as libc is also resolved in the host and not the compat layer

{EESSI 2023.06} $ /usr/bin/ldd HelloWorld
./HelloWorld: /lib64/libc.so.6: version `GLIBC_2.36' not found (required by /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/libevent/2.1.12-GCCcore-13.2.0/lib64/libevent_core-2.1.so.7)
./HelloWorld: /lib64/libc.so.6: version `GLIBC_ABI_DT_RELR' not found (required by /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib/../lib64/libm.so.6)
linux-vdso.so.1 (0x00007fffe4fd3000)
libmpi.so.40 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/OpenMPI/4.1.6-GCC-13.2.0/lib/libmpi.so.40 (0x00007f1fdf571000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1fdf200000)
[...]
libevent_pthreads-2.1.so.7 => /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/intel/skylake_avx512/software/libevent/2.1.12-GCCcore-13.2.0/lib64/libevent_pthreads-2.1.so.7 (0x00007f1fdf420000)
libm.so.6 => /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib/../lib64/libm.so.6 (0x00007f1fdeeb1000)
/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f1fdf698000)

```
Now is the moment of truth, if everything looks right when checking with ldd, you should be fine to run the program:

```
{EESSI 2023.06} $ mpirun -n 2 HelloWorld
Hello World from Node 0
Hello World from Node 1

```
Even when closing the shell and restarting the environment, the libraries should point to the directories we set in `LD_RUN_PATH`, still, remember to load the required dependencies before running the binary.

!!! warning

Expand Down