J uranic/codex dirschema fix

CHANGELOG.md

@@ -7,6 +7,7 @@
 - Update Segmentation masks directory schema
 - Update Auto-fluorescence directory schema
 - Update CODEX directory schema
+- Update CODEX directory schema
 
 ## v0.0.28
 - Update Xenium directory schema

docs/codex/current/index.md

@@ -28,7 +28,31 @@ Related files:
 <br>
 
 ## Directory schemas
-<summary><b>Version 2.1 (use this one)</b></summary>
+<summary><b>Version 2.2 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | This is a directory containing raw data. |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/images\/.*</code> | ✓ | This is a directory containing processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> | ✓ | OME-TIFF file (multichannel, multi-layered) produced by the experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file should describe any processing that was done to generate the images in each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. Two columns should be booleans "is this a channel to use for nuclei segmentation"  and "is this a channel to use for cell segmentation". |
+| <code>[^\/]*NAV[^\/]*\.tif</code> (example: <code>NAV.tif</code>) |  | Navigational Image showing Region of Interest (Keyance Microscope only) |
+| <code>[^\/]+\.pdf</code> (example: <code>summary.pdf</code>) |  | **[QA/QC]** PDF export of Powerpoint slide deck containing the Image Analysis Report |
+| <code>extras\/dir-schema-v2-with-dataset-json</code> | ✓ | Empty file whose presence indicates the version of the directory schema in use |
+| <code>processed\/drv_[^\/]*\/.*</code> | ✓ | Processed files produced by the Akoya software or alternative software. |
+| <code>raw\/cyc[^\/]*_reg[^\/]*\/.*</code> | ✓ | Intermediary directory |
+| <code>raw\/src_[^\/]*\/.*</code> | ✓ | Intermediary directory |
+| <code>raw\/cyc[^\/]*_reg[^\/]*\/[^\/]*_z[^\/]*_CH[^\/]*\.tif</code> | ✓ | TIFF files produced by the experiment. General folder format: Cycle(n)_Region(n)_date; General file format: name_tileNumber(n)_zplaneNumber(n)_channelNumber(n) |
+| <code>raw\/src_[^\/]*\/cyc[^\/]*_reg[^\/]*_[^\/]*\/[^\/]+\.gci</code> |  | Group Capture Information File (Keyance Microscope only) |
+| <code>raw\/dataset\.json</code> (example: <code>raw/dataset.json</code>) | ✓ | Data processing parameters file. This will include additional CODEX specific metadata needed for the HIVE processing workflow. |
+| <code>raw\/reg_[^\/]*\.png</code> (example: <code>raw/reg_00.png</code>) |  | Region overviews |
+| <code>raw\/experiment\.json</code> (example: <code>raw/experiment.json</code>) |  | JSON file produced by the Akoya software which contains the metadata for the experiment, including the software version used, microscope parameters, channel names, pixel dimensions, etc. (required for HuBMAP pipeline) |
+
+<summary><b>Version 2.1</b></summary>
 
 | pattern | required? | description |
 | --- | --- | --- |

src/ingest_validation_tools/directory-schemas/codex-v2.2.yaml

@@ -0,0 +1,88 @@
+files:
+  -
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset. [Exists in all assays]
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -  
+    pattern: raw\/.*
+    required: True
+    description: This is a directory containing raw data.
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: This is a directory containing processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF file (multichannel, multi-layered) produced by the experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file should describe any processing that was done to generate the images in each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. Two columns should be booleans "is this a channel to use for nuclei segmentation"  and "is this a channel to use for cell segmentation".
+  -
+    pattern: '[^\/]*NAV[^\/]*\.tif'
+    required: False
+    description: Navigational Image showing Region of Interest (Keyance Microscope only)
+    is_qa_qc: False
+    example: NAV.tif
+  -
+    pattern: '[^\/]+\.pdf'
+    required: False
+    description: PDF export of Powerpoint slide deck containing the Image Analysis Report
+    is_qa_qc: True
+    example: summary.pdf
+  -
+    pattern: extras\/dir-schema-v2-with-dataset-json
+    required: True
+    description: Empty file whose presence indicates the version of the directory schema in use
+    is_qa_qc: False
+  -
+    pattern: processed\/drv_[^\/]*\/.*
+    required: True
+    description: Processed files produced by the Akoya software or alternative software.
+  -
+    pattern: raw\/cyc[^\/]*_reg[^\/]*\/.*
+    required: True
+    description: Intermediary directory
+  -
+    pattern: raw\/src_[^\/]*\/.*
+    required: True
+    description: Intermediary directory
+  -
+    pattern: raw\/cyc[^\/]*_reg[^\/]*\/[^\/]*_z[^\/]*_CH[^\/]*\.tif
+    required: True
+    description: 'TIFF files produced by the experiment. General folder format: Cycle(n)_Region(n)_date; General file format: name_tileNumber(n)_zplaneNumber(n)_channelNumber(n)'
+  -
+    pattern: raw\/src_[^\/]*\/cyc[^\/]*_reg[^\/]*_[^\/]*\/[^\/]+\.gci
+    required: False
+    description: Group Capture Information File (Keyance Microscope only)
+  -
+    pattern: raw\/dataset\.json
+    required: True
+    description: Data processing parameters file. This will include additional CODEX specific metadata needed for the HIVE processing workflow. 
+    example: raw/dataset.json
+  -
+    pattern: raw\/reg_[^\/]*\.png
+    required: False
+    description: Region overviews
+    example: raw/reg_00.png
+  -
+    pattern: raw\/experiment\.json
+    required: False
+    description: JSON file produced by the Akoya software which contains the metadata for the experiment, including the software version used, microscope parameters, channel names, pixel dimensions, etc. (required for HuBMAP pipeline) 
+    example: raw/experiment.json