From 77e73ee29510c91ce0d7e183d49289b07ce5c8f5 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 12 Nov 2024 12:51:16 +0100 Subject: [PATCH 1/3] Add singularity.libraryDir to config page Signed-off-by: Christopher Hakkaart --- docs/reference/config.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/config.md b/docs/reference/config.md index 464ca480ba..e386c9be12 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -1375,6 +1375,9 @@ The following settings are available: `singularity.envWhitelist` : Comma separated list of environment variable names to be included in the container environment. +`singularity.libraryDir` +: Directory where remote Singularity images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. + `singularity.noHttps` : Pull the Singularity image with http protocol (default: `false`). From 0ce806ecbf69b7bb1cc4b334dfa6f19be8ac97fc Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 12 Nov 2024 20:07:15 +0100 Subject: [PATCH 2/3] Update Apptainer and add versioned notes Signed-off-by: Christopher Hakkaart --- docs/container.md | 4 ++-- docs/reference/config.md | 11 +++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/container.md b/docs/container.md index 1d0ad99e51..9cbe208ed3 100644 --- a/docs/container.md +++ b/docs/container.md @@ -123,7 +123,7 @@ This feature requires the `apptainer` tool to be installed where the workflow ex Nextflow caches those images in the `apptainer` directory in the pipeline work directory by default. However it is suggested to provide a centralised cache directory by using either the `NXF_APPTAINER_CACHEDIR` environment variable or the `apptainer.cacheDir` setting in the Nextflow config file. :::{versionadded} 21.09.0-edge -When looking for a Apptainer image file, Nextflow first checks the *library* directory, and if the image file is not found, the *cache* directory is used s usual. The library directory can be defined either using the `NXF_APPTAINER_LIBRARYDIR` environment variable or the `apptainer.libraryDir` configuration setting (the latter overrides the former). +Nextflow first checks the library directory when retrieving Apptainer images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_APPTAINER_LIBRARYDIR` environment variable. The `apptainer.libraryDir` configuration option overrides `NXF_APPTAINER_LIBRARYDIR` environment variable if both are set. ::: :::{warning} @@ -652,7 +652,7 @@ The `library://` pseudo-protocol allows you to import Singularity images from a Nextflow caches the images in `${NXF_WORK}/singularity` by default. However, it is recommended to define a centralised cache directory using either the `NXF_SINGULARITY_CACHEDIR` environment variable or the `singularity.cacheDir` setting in the Nextflow config file. :::{versionadded} 21.09.0-edge -When looking for a Singularity image file, Nextflow first checks the *library* directory, and if the image file is not found, the *cache* directory is used as usual. The library directory can be defined either using the `NXF_SINGULARITY_LIBRARYDIR` environment variable or the `singularity.libraryDir` configuration setting (the latter overrides the former). +Nextflow first checks the library directory when retrieving Singularity images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The `singularity.libraryDir` configuration option overrides `NXF_SINGULARITY_LIBRARYDIR` environment variable if both are set. ::: :::{warning} diff --git a/docs/reference/config.md b/docs/reference/config.md index e386c9be12..4d92dd31b7 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -52,6 +52,13 @@ The following settings are available: `apptainer.envWhitelist` : Comma separated list of environment variable names to be included in the container environment. +`apptainer.libraryDir` +: Directory where remote Apptainer images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. + + :::{versionadded} 21.09.0-edge + Nextflow first checks the library directory when retrieving Apptainer images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_APPTAINER_LIBRARYDIR` environment variable. The `apptainer.libraryDir` configuration option overrides `NXF_APPTAINER_LIBRARYDIR` environment variable if both are set. + ::: + `apptainer.noHttps` : Pull the Apptainer image with http protocol (default: `false`). @@ -1378,6 +1385,10 @@ The following settings are available: `singularity.libraryDir` : Directory where remote Singularity images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. + :::{versionadded} 21.09.0-edge + Nextflow first checks the library directory when retrieving Singularity images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The `singularity.libraryDir` configuration option overrides `NXF_SINGULARITY_LIBRARYDIR` environment variable if both are set. + ::: + `singularity.noHttps` : Pull the Singularity image with http protocol (default: `false`). From e84be328b4573183e90fa8b59fbddc8f5e6625ad Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 26 Nov 2024 12:38:29 +0100 Subject: [PATCH 3/3] Changed description as per suggestion, removed date, removed notes in reference Signed-off-by: Christopher Hakkaart --- docs/container.md | 20 ++++++++++---------- docs/reference/config.md | 8 -------- 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/docs/container.md b/docs/container.md index 363e0aae58..ba8a563828 100644 --- a/docs/container.md +++ b/docs/container.md @@ -94,7 +94,7 @@ Read the {ref}`Process scope ` section to learn more about proce Nextflow is able to transparently pull remote container images stored in any Docker compatible registry. -By default when a container name is specified, Nextflow checks if an image file with that name exists in the local file system. If that image file exists, it's used to execute the container. If a matching file does not exist, Nextflow automatically tries to pull an image with the specified name from the container registry. +By default, when a container name is specified, Nextflow checks if an image file with that name exists in the local file system. If that image file exists, it's used to execute the container. If a matching file does not exist, Nextflow automatically tries to pull an image with the specified name from the container registry. If you want Nextflow to check only for local file images, prefix the container name with the `file://` pseudo-protocol. For example: @@ -107,7 +107,7 @@ apptainer.enabled = true Use three `/` slashes to specify an **absolute** file path, otherwise the path will be interpreted as relative to the workflow launch directory. ::: -To pull images from Apptainer Hub or a third party Docker registry, simply prefix the image name with the `shub://`, `docker://` or `docker-daemon://` pseudo-protocol as required by Apptainer. For example: +To pull images from Apptainer Hub or a third party Docker registry, prefix the image name with the `shub://`, `docker://` or `docker-daemon://` pseudo-protocol as required by Apptainer. For example: ```groovy process.container = 'docker://quay.io/biocontainers/multiqc:1.3--py35_2' @@ -120,11 +120,11 @@ You do not need to specify `docker://` to pull from a Docker repository. Nextflo This feature requires the `apptainer` tool to be installed where the workflow execution is launched (as opposed to the compute nodes). ::: -Nextflow caches those images in the `apptainer` directory in the pipeline work directory by default. However it is suggested to provide a centralised cache directory by using either the `NXF_APPTAINER_CACHEDIR` environment variable or the `apptainer.cacheDir` setting in the Nextflow config file. +Nextflow caches Apptainer images in the `apptainer` directory, in the pipeline work directory, by default. However, it is recommended to provide a centralized cache directory using the `NXF_APPTAINER_CACHEDIR` environment variable or the `apptainer.cacheDir` setting in the Nextflow config file. -:::{versionadded} 21.09.0-edge -Nextflow first checks the library directory when retrieving Apptainer images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_APPTAINER_LIBRARYDIR` environment variable. The `apptainer.libraryDir` configuration option overrides `NXF_APPTAINER_LIBRARYDIR` environment variable if both are set. -::: +Nextflow uses the library directory to determine the location of Apptainer containers. The library directory can be defined using the `apptainer.libraryDir` configuration setting or the `NXF_APPTAINER_LIBRARYDIR` environment variable. The configuration file option overrides the environment variable if both are set. + +Nextflow first checks the library directory when searching for the image. If the image is not found it then checks the cache directory. The main difference between the library directory and the cache directory is that the first is assumed to be a read-only container repository, while the latter is expected to be writable path where container images can added for caching purposes. :::{warning} When using a compute cluster, the Apptainer cache directory must reside in a shared filesystem accessible to all compute nodes. @@ -653,11 +653,11 @@ process.container = 'library://library/default/alpine:3.8' The `library://` pseudo-protocol allows you to import Singularity images from a local Docker installation instead of downloading them from a Docker registry. This feature requires the `singularity` tool to be installed where the workflow execution is launched (as opposed to the compute nodes). -Nextflow caches the images in `${NXF_WORK}/singularity` by default. However, it is recommended to define a centralised cache directory using either the `NXF_SINGULARITY_CACHEDIR` environment variable or the `singularity.cacheDir` setting in the Nextflow config file. +Nextflow caches Singularity images in the `singularity` directory, in the pipeline work directory, by default. However, it is recommended to provide a centralized cache directory using the `NXF_SINGULARITY_CACHEDIR` environment variable or the `singularity.cacheDir` setting in the Nextflow config file. -:::{versionadded} 21.09.0-edge -Nextflow first checks the library directory when retrieving Singularity images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The `singularity.libraryDir` configuration option overrides `NXF_SINGULARITY_LIBRARYDIR` environment variable if both are set. -::: +Nextflow uses the library directory to determine the location of Singularity images. The library directory can be defined using the `singularity.libraryDir` configuration setting or the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The configuration file option overrides the environment variable if both are set. + +Nextflow first checks the library directory when searching for the image. If the image is not found it then checks the cache directory. The main difference between the library directory and the cache directory is that the first is assumed to be a read-only container repository, while the latter is expected to be writable path where container images can added for caching purposes. :::{warning} When using a compute cluster, the Singularity cache directory must reside in a shared filesystem accessible to all compute nodes. diff --git a/docs/reference/config.md b/docs/reference/config.md index 4d92dd31b7..0efb1ce6ae 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -55,10 +55,6 @@ The following settings are available: `apptainer.libraryDir` : Directory where remote Apptainer images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. - :::{versionadded} 21.09.0-edge - Nextflow first checks the library directory when retrieving Apptainer images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_APPTAINER_LIBRARYDIR` environment variable. The `apptainer.libraryDir` configuration option overrides `NXF_APPTAINER_LIBRARYDIR` environment variable if both are set. - ::: - `apptainer.noHttps` : Pull the Apptainer image with http protocol (default: `false`). @@ -1385,10 +1381,6 @@ The following settings are available: `singularity.libraryDir` : Directory where remote Singularity images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. - :::{versionadded} 21.09.0-edge - Nextflow first checks the library directory when retrieving Singularity images. The cache directory is used if the image file is not found. The library directory can also be defined using the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The `singularity.libraryDir` configuration option overrides `NXF_SINGULARITY_LIBRARYDIR` environment variable if both are set. - ::: - `singularity.noHttps` : Pull the Singularity image with http protocol (default: `false`).