Skip to content

Commit

Permalink
better mermaid
Browse files Browse the repository at this point in the history
  • Loading branch information
fstagni committed Oct 18, 2024
1 parent f2fc004 commit 00925fe
Showing 1 changed file with 121 additions and 46 deletions.
167 changes: 121 additions & 46 deletions slides.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,39 +122,75 @@ layout: side-title
align: lm-lm
color: gray-light
title: WMS
titlewidth: is-3
---

:: title ::

# WMS
### Workload Management System
## Workload Management System
- Pull model based on Pilot jobs
- Also "Push" solution for HPCs that do not support pilots (because of limited internet access).
- Will integrate [CWL (Common Workflow Language)](https://www.commonwl.org) as a way of defining jobs (replacing JDL) --> see poster #217

:: content ::

```mermaid
%%{init: { 'theme': 'default' }}%%
flowchart LR;
WMS((WMS))
Users-->ProductionSystem;
ProductionSystem-->TransformationSystem;
TransformationSystem-->WMS;
Users-->CLI;
Users-->API;
Users-->WebApp;
CLI-->WMS;
API-->WMS;
WebApp-->WMS;
WMS-->HTCondorCE;
WMS-->ARC-AREX;
WMS-->IaaS:Clouds;
WMS-->SSH;
SSH-->HTCondor;
SSH-->SLURM;
WMS-->HPCs;
HPCs-->SLURM;
style WMS fill:#bbf
WMS[("`**Workload
Management
System**`")]
style WMS fill:#bbf
TS[("`**Transformation
System**`")]
PS[("`**Production
System**`")]
HPC["`High
Perfomance
Computers`"]
style HPC fill:#A145
clusters["`Computer clusters`"]
style clusters fill:#A145
CLI["`**CLI**`"]
API["`**APIs**`"]
WebApp["`**WebApp**`"]
Grid_Nodes["Grid"]
style HTCondorCE fill:#F23
style ARC-AREX fill:#F23
style libcloud fill:#F23
style SSH fill:#F23
style Grid_Nodes fill:#A145
style Iaas:Clouds fill:#A145
style HTCondor fill:#F26
style SLURM fill:#F26
Users-->CLI
Users-->API
Users-->WebApp
Users-->PS
PS-->TS
CLI-->|jobs|WMS
API-->|jobs|WMS
WebApp-->|jobs|WMS
TS-->|jobs|WMS
WMS-->|pilots|libcloud
WMS-->|pilots|HTCondorCE
WMS-->|pilots|ARC-AREX
WMS-->|pilots|SSH
libcloud-->|VMs starting pilots|Iaas:Clouds
HTCondorCE-->Grid_Nodes
ARC-AREX-->Grid_Nodes
ARC-AREX-->HPC
SSH-->|pilots|SLURM
SSH-->|pilots|HTCondor
SSH-->|pilots|clusters
SLURM-->HPC
SLURM-->clusters
HTCondor-->clusters
```

---
Expand All @@ -167,8 +203,7 @@ titlewidth: is-5

:: title ::

# DMS
### Data Management System
## Data Management System
It’s about **files**:​ placing, replicating, removing files​

- there are **LFNs** (logical file names)
Expand All @@ -180,17 +215,44 @@ It’s about **files**:​ placing, replicating, removing files​
:: content ::

```mermaid
%%{init: { 'theme': 'default' }}%%
flowchart LR;
DataManager((DataManager))
FileCatalog-->DFC_Service;
FileCatalog-->Rucio;
FileCatalog-->Transformation_Service;
DataManager-->FileCatalog;
DataManager-->StorageBase;
StorageBase-->SE1;
StorageBase-->SE2;
StorageBase-->SE3;
style FileCatalog fill:#bbf
RMS[("`**Request
Management
System**`")]
DMS[("`**Data
Management
System**`")]
style DMS fill:#bbf
FC[["`**Files
Catalog**`"]]
style FC fill:#bbf
StorageBase[["`**Storage Base**`"]]
style StorageBase fill:#bbf
DFC[("`DIRAC
Files
Catalog`")]
Rucio[("Rucio")]
style Rucio fill:#6001
TS[("`DIRAC
Transformation
System`")]
WebDav["WebDav (http)"]
style WebDav fill:#F23
style XRootD fill:#F23
style SRM fill:#F23
Users-->DMS
RMS-->DMS
DMS-->FC
DMS-->StorageBase
FC-->DFC
FC-->Rucio
FC-->TS
StorageBase-->WebDav
StorageBase-->XRootD
StorageBase-->SRM
```


Expand All @@ -203,7 +265,7 @@ title: TS

:: title ::

# TS (Transformation System)
## Transformation System
### For productions and Dataset management

- A *Data Processing* **transformation** (e.g. Simulation, Merge, DataReconstruction...) creates jobs in the WMS (and re-submit them if needed, eventually destroy them).​
Expand All @@ -213,25 +275,38 @@ title: TS
:: content ::

<span class="bg-cyan-100 text-cyan-600 p-4 border-l-6 border-2 border-cyan-400 rounded-lg pl-8 pr-8 w-full block">
The TS is used to automate common tasks related to production activities. It can handle thousands of productions, millions of files and jobs.
The Transformation System is used to automate common tasks related to production activities. It can handle thousands of productions, millions of files and jobs.
</span>

&nbsp;
&nbsp;

```mermaid
%%{init: { 'theme': 'default' }}%%
flowchart LR;
TransformationSystem((TransformationSystem))
Productions_Management-->TransformationSystem;
DataSets_Management-->TransformationSystem;
TransformationSystem-->WorkloadManagementSystem;
WorkloadManagementSystem-->Jobs;
WorkloadManagementSystem-->Sites;
TransformationSystem-->RequestManagementSystem;
RequestManagementSystem-->DMS;
DMS-->Catalogs;
DMS-->SEs;
style TransformationSystem fill:#bbf
TS[("`**Transformation
System**`")]
style TS fill:#bbf
WMS[("`**Workload
Management
System**`")]
style WMS fill:#bba
RMS[("`**Request
Management
System**`")]
style RMS fill:#bba
DMS[("`**Data
Management
System**`")]
style DMS fill:#bba
PM["Productions Management"]
DM["DataSets Management"]
PM-->TS
DM-->TS
TS-->WMS
TS-->RMS
RMS-->DMS
```

---
Expand Down

0 comments on commit 00925fe

Please sign in to comment.