Skip to content

Commit

Permalink
Cite DMTN-213
Browse files Browse the repository at this point in the history
  • Loading branch information
airnandez committed Jan 15, 2025
1 parent 1feb7a3 commit d22a09f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion DMTN-306.tex
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ \section{Data movement tools}

Rubin Observatory operates a dedicated instance of Rucio, configured to transfer files between the Rucio Storage Elements (RSEs) at each facility. These storage endpoints support a data movement protocol that Rucio utilizes to transport data across them. The US and UK data facilities use XrootD \cite{xrootd}, while the France data facility uses dCache \cite{dCache}. All of these systems expose the webDAV protocol \cite{webdav}, an extension of HTTP \cite{http1.1}. Data is transferred securely across sites using confidential channels built on top of secure HTTP.

Each processing facility exposes at least two RSEs: one for storing input data required for processing (e.g., raw images, calibration data, reference catalogs, etc.) and another endpoint for storing the products generated by the image processing pipelines. Data stored by the input data RSE is protected against modification and removal and is also archived to tape. Data products stored in the products RSE are less sensitive as they can be regenerated and even some of them may be deleted after a processing campaign is complete. All RSEs are configured to use the identity logical-to-physical filename mapping. This configuration ensures that the file pathnames are preserved relative to the Butler repository's datastore location, which is critical for proper file replication to the destination where the Butler expects to find them.
Each processing facility exposes at least two RSEs: one for storing input data required for processing (e.g., raw images, calibration data, reference catalogs, etc.) and another endpoint for storing the products generated by the image processing pipelines \cite{DMTN-213}. Data stored by the input data RSE is protected against modification and removal and is also archived to tape. Data products stored in the products RSE are less sensitive as they can be regenerated and even some of them may be deleted after a processing campaign is complete. All RSEs are configured to use the identity logical-to-physical filename mapping. This configuration ensures that the file pathnames are preserved relative to the Butler repository's datastore location, which is critical for proper file replication to the destination where the Butler expects to find them.

\subsection{Registration of files to replicate}
\label{registration}
Expand Down

0 comments on commit d22a09f

Please sign in to comment.