-
Notifications
You must be signed in to change notification settings - Fork 35
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #188 from casparvl/host_injections
Host injections, lmod hooks and moving gpu-related host-injection instructions
- Loading branch information
Showing
10 changed files
with
268 additions
and
52 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # How to configure EESSI | ||
|
|
||
| ## Why configuration is necessary | ||
|
|
||
| Just [installing EESSI](../getting_access/native_installation.md) is enough to get started with the EESSI software stack on a CPU-based system. However, additional configuration is necessary in many other cases, such as | ||
| - enabling GPU support on GPU-based systems | ||
| - site-specific configuration / tuning of the MPI libraries provided by EESSI | ||
| - overriding EESSI's MPI library with an ABI compatible host MPI | ||
|
|
||
| ## The `host_injections` variant symlink | ||
|
|
||
| To allow such site-specific configuration, the EESSI repository includes a special directory where system administrations can install files that can be picked up by the software installations included in EESSI. This special directory is located in `/cvmfs/software.eessi.io/host_injections`, and it is a *CernVM-FS Variant Symlink*: | ||
| a symbolic link for which the target can be controlled by the CernVM-FS client configuration (for more info, see ['Variant Symlinks' in the official CernVM-FS documentation](https://cvmfs.readthedocs.io/en/stable/cpt-repo.html#variant-symlinks)). | ||
|
|
||
| !!! info "Default target for `host_injections` variant symlink" | ||
|
|
||
| Unless otherwise configured in the CernVM-FS client configuration for the EESSI repository, the `host_injections` symlink points to `/opt/eessi` on the client system: | ||
| ``` | ||
| $ ls -l /cvmfs/software.eessi.io/host_injections | ||
| lrwxrwxrwx 1 cvmfs cvmfs 10 Oct 3 13:51 /cvmfs/software.eessi.io/host_injections -> /opt/eessi | ||
| ``` | ||
|
|
||
| The target for this symlink can be controlled by setting the `EESSI_HOST_INJECTIONS` variable in your local CVMFS configuration for EESSI. E.g. | ||
| ```{bash} | ||
| sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/' > /etc/cvmfs/domain.d/eessi.io.local" | ||
| ``` | ||
|
|
||
| !!! note "Don't forget to reload the CernVM-FS configuration" | ||
| After making a change to a CernVM-FS configuration file, you also need to reload the configuration: | ||
| ```{ .bash .copy } | ||
| sudo cvmfs_config reload | ||
| ``` | ||
|
|
||
| On a heterogeneous system, you may want to use different targets for the variant symlink for different node types. For example, you might have two types of GPU nodes (`gpu1` and `gpu2`) for which the GPU drivers are _not_ in the same location, or not of the same version. Since those are both things we configure under `host_injections`, you'll need separate `host_injections` directories for each node type. That can easily be achieved by putting e.g. | ||
|
|
||
| ```{bash} | ||
| sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/gpu1/' > /etc/cvmfs/domain.d/eessi.io.local" | ||
| ``` | ||
|
|
||
| in the CVMFS config on the `gpu1` nodes, and | ||
|
|
||
| ```{bash} | ||
| sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/gpu2/' > /etc/cvmfs/domain.d/eessi.io.local" | ||
| ``` | ||
| in the CVMFS config on the `gpu2` nodes. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| # Configuring site-specific Lmod hooks | ||
| You may want to customize what happens when certain modules are loaded, for example, you may want to set additional environment variables. This is possible with [LMOD hooks](https://lmod.readthedocs.io/en/latest/170_hooks.html). A typical example would be when you want to tune the OpenMPI module for your system by setting additional environment variables when an OpenMPI module is loaded. | ||
|
|
||
|
|
||
| ## Location of the hooks | ||
| The EESSI software stack provides its own set of hooks in `$LMOD_PACKAGE_PATH/SitePackage.lua`. This `SitePackage.lua` also searches for site-specific hooks in two additional locations: | ||
|
|
||
| - `$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/.lmod/SitePackage.lua` | ||
| - `$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/software/$EESSI_OS_TYPE/$EESSI_SOFTWARE_SUBDIR/.lmod/SitePackage.lua` | ||
|
|
||
| The first allows for hooks that need to be executed for that system, irrespective of the CPU architecture. The second allows for hooks specific to a certain architecture. | ||
|
|
||
| ## Architecture-independent hooks | ||
| Hooks are written in Lua and can use any of the standard Lmod functionality as described in the [Lmod documentation](https://lmod.readthedocs.io/en/latest/170_hooks.html). While there are many types of hooks, you most likely want to specify a load or unload hook. Note that the EESSI hooks provide a nice example of what you can do with hooks. Here, as an example, we will define a `load` hook that environment variable `MY_ENV_VAR` to `1` whenever an `OpenMPI` module is loaded. | ||
|
|
||
| First, you typically want to load the necessary Lua packages: | ||
| ```lua | ||
| -- $EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/.lmod/SitePackage.lua | ||
|
|
||
| -- The Strict package checks for the use of undeclared variables: | ||
| require("strict") | ||
|
|
||
| -- Load the Lmod Hook package | ||
| local hook=require("Hook") | ||
| ``` | ||
|
|
||
| Next, we define a function that we want to use as a hook. Unfortunately, registering multiple hooks of the same type (e.g. multiple `load` hooks) is only supported in Lmod 8.7.35+. EESSI version 2023.06 uses Lmod 8.7.30. Thus, we define our function without the local keyword, so that we can still add to it later in an architecture-specific hook (if we wanted to): | ||
|
|
||
| ```lua | ||
| -- Define a function for the hook | ||
| -- Note that we define this without 'local' keyword | ||
| -- That way we can still add to this function in an architecture-specific hook | ||
| function set_my_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_ENV_VAR', '1') | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| for the same reason that multiple hooks cannot be registered, we need to combine this function for our site-specific (architecture-independent) with the function that specifies the EESSI `load` hook. Note that all EESSI hooks will be called `eessi_<hook_type>_hook` by convention. | ||
|
|
||
| ```lua | ||
| -- Registering multiple hook functions, e.g. multiple load hooks is only supported in Lmod 8.7.35+ | ||
| -- EESSI version 2023.06 uses lmod 8.7.30. Thus, we first have to combine all functions into a single one, | ||
| -- before registering it as a hook | ||
| local function combined_load_hook(t) | ||
| -- Call the EESSI load hook (if it exists) | ||
| -- Note that if you wanted to overwrite the EESSI hooks (not recommended!), you would omit this | ||
| if eessi_load_hook ~= nil then | ||
| eessi_load_hook(t) | ||
| end | ||
| -- Call the site-specific load hook | ||
| set_my_env_var_openmpi(t) | ||
| end | ||
| ``` | ||
|
|
||
| Then, we can finally register this function as an Lmod hook: | ||
|
|
||
| ```lua | ||
| hook.register("load", combined_load_hook) | ||
| ``` | ||
|
|
||
| Thus, our complete `$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/.lmod/SitePackage.lua` now looks like this (omitting the comments): | ||
|
|
||
| ```lua | ||
| require("strict") | ||
| local hook=require("Hook") | ||
|
|
||
| function set_my_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_ENV_VAR', '1') | ||
| end | ||
| end | ||
|
|
||
| local function combined_load_hook(t) | ||
| if eessi_load_hook ~= nil then | ||
| eessi_load_hook(t) | ||
| end | ||
| set_my_env_var_openmpi(t) | ||
| end | ||
|
|
||
| hook.register("load", combined_load_hook) | ||
| ``` | ||
|
|
||
| Note that for future EESSI versions, if they use Lmod 8.7.35+, this would be simplified to: | ||
|
|
||
| ```lua | ||
| require("strict") | ||
| local hook=require("Hook") | ||
|
|
||
| local function set_my_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_ENV_VAR', '1') | ||
| end | ||
| end | ||
|
|
||
| hook.register("load", set_my_env_var_openmpi, "append") | ||
| ``` | ||
|
|
||
| ## Architecture-dependent hooks | ||
| Now, assume that in addition we want to set an environment variable `MY_SECOND_ENV_VAR` to `5`, but only for nodes that have the `zen3` architecture. First, again, you typically want to load the necessary Lua packages: | ||
|
|
||
| ```lua | ||
| -- $EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua | ||
|
|
||
| -- The Strict package checks for the use of undeclared variables: | ||
| require("strict") | ||
|
|
||
| -- Load the Lmod Hook package | ||
| local hook=require("Hook") | ||
| ``` | ||
|
|
||
| Next, we define the function for the hook itself | ||
|
|
||
| ```lua | ||
| -- Define a function for the hook | ||
| -- This time, we can define it as a local function, as there are no hooks more specific than this | ||
| local function set_my_second_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_SECOND_ENV_VAR', '5') | ||
| end | ||
| end | ||
| ``` | ||
|
|
||
| Then, we combine the functions into one | ||
|
|
||
| ```lua | ||
| local function combined_load_hook(t) | ||
| -- Call the EESSI load hook first | ||
| if eessi_load_hook ~= nil then | ||
| eessi_load_hook(t) | ||
| end | ||
| -- Then call the architecture-independent load hook | ||
| if set_my_env_var_openmpi(t) ~= nil then | ||
| set_my_env_var_openmpi(t) | ||
| end | ||
| -- And finally the architecture-dependent load hook we just defined | ||
| set_my_second_env_var_openmpi(t) | ||
| end | ||
| ``` | ||
|
|
||
| before finally registering it as an Lmod hook | ||
|
|
||
| ```lua | ||
| hook.register("load", combined_load_hook) | ||
| ``` | ||
|
|
||
| Thus, our full `$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/software/linux/x86_64/amd/zen3/.lmod/SitePackage.lua` now looks like this (omitting the comments): | ||
|
|
||
| ```lua | ||
| require("strict") | ||
| local hook=require("Hook") | ||
|
|
||
| local function set_my_second_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_SECOND_ENV_VAR', '5') | ||
| end | ||
| end | ||
|
|
||
| local function combined_load_hook(t) | ||
| if eessi_load_hook ~= nil then | ||
| eessi_load_hook(t) | ||
| end | ||
| if set_my_env_var_openmpi(t) ~= nil then | ||
| set_my_env_var_openmpi(t) | ||
| end | ||
| set_my_second_env_var_openmpi(t) | ||
| end | ||
|
|
||
| hook.register("load", combined_load_hook) | ||
| ``` | ||
|
|
||
| Again, note that for future EESSI versions, if they use Lmod 8.7.35+, this would simplify to | ||
|
|
||
| ```lua | ||
| require("strict") | ||
| local hook=require("Hook") | ||
|
|
||
| local function set_my_second_env_var_openmpi(t) | ||
| local simpleName = string.match(t.modFullName, "(.-)/") | ||
| if simpleName == 'OpenMPI' then | ||
| setenv('MY_SECOND_ENV_VAR', '5') | ||
| end | ||
| end | ||
|
|
||
| hook.register("load", set_my_second_var_openmpi, "append") | ||
| ``` |
Oops, something went wrong.