Juranic/xenium dirschema

CHANGELOG.md

@@ -9,6 +9,7 @@
 - Update GeoMx NGS directory schema
 - Update shared upload check
 - Release Xenium
+- Create Xenium directory schema
 
 ## v0.0.26
 - Update GeoMx NGS directory schema

docs/xenium/current/index.md

@@ -28,5 +28,25 @@ Related files:
 <br>
 
 ## Directory schemas
-<summary><a href="https://docs.google.com/spreadsheets/d/1LE-iyY2E6eP4E8jhgP6rhsvjESrdHXWYrMwKTvNkI5Y"><b>Version 2.0 (use this one)</b> (draft - submission of data prepared using this schema will be supported by Sept. 30) </a></summary>
+<summary><b>Version 2.0 (use this one)</b></summary>
+
+| pattern | required? | description |
+| --- | --- | --- |
+| <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. |
+| <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>extras\/microscope_settings\.json</code> |  | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. |
+| <code>raw\/.*</code> | ✓ | All raw data files for the experiment. |
+| <code>raw\/markers\.csv</code> |  | A csv file describing any morphology markers used to guide ROI and/or AOI selection. |
+| <code>raw\/additional_panels_used\.csv</code> |  | If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code. |
+| <code>raw\/custom_probe_set\.csv</code> |  | This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>). |
+| <code>raw\/custom_probe_set\.bed</code> |  | This is a BED file version of the custom probe set file. |
+| <code>raw\/transcript_locations\.csv</code> | ✓ | The origin of the coordinate is 0,0 at the top left corner of the image. The file should include: gene name, x, y, z (optional), quality score (optional). It is expected that the first row in the file contains the column header. |
+| <code>raw\/custom_gene_list\.csv</code> |  | This describes the target genes profiled by the assay. For advanced design, this can be probes sequences for splicing or other analysis for any target of interest. The format should minimally contain: gene name, ensemble ID |
+| <code>raw\/probes\.csv</code> |  | A CSV file describing the probe panel used. This is tyipcally what's used to specific the probe set when ordering a probe panel for a Xenium run. |
+| <code>raw\/gene_panel\.json</code> | ✓ | This is the JSON file describing the probes, as output from the xenium-ranger pipeline. |
+| <code>raw\/images\/overlay\.(?:jpeg&#124;tiff)</code> |  | State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file |
+| <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. |
+| <code>lab_processed\/images\/.*</code> | ✓ | Processed image files |
+| <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> |
+| <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> |
 

src/ingest_validation_tools/directory-schemas/xenium-v2.0.yaml

@@ -1,4 +1,74 @@
-draft: true
 files:
   -
-    draft_link: 'https://docs.google.com/spreadsheets/d/1LE-iyY2E6eP4E8jhgP6rhsvjESrdHXWYrMwKTvNkI5Y'
+    pattern: extras\/.*
+    required: True
+    description: Folder for general lab-specific files related to the dataset.
+  -
+    pattern: extras\/microscope_hardware\.json
+    required: True
+    description: A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: extras\/microscope_settings\.json
+    required: False
+    description: A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document.
+    is_qa_qc: True
+  -
+    pattern: raw\/.*
+    required: True
+    description: All raw data files for the experiment.
+  -
+    pattern: raw\/markers\.csv
+    required: False
+    description: A csv file describing any morphology markers used to guide ROI and/or AOI selection.
+  -
+    pattern: raw\/additional_panels_used\.csv
+    required: False
+    description: If multiple commercial probe panels were used, then the primary probe panel should be selected in the "oligo_probe_panel" metadata field. The additional panels must be included in this file. Each panel record should include:manufacturer, model/name, product code.
+  -
+    pattern: raw\/custom_probe_set\.csv
+    required: False
+    description: This file should contain any custom probes used and must be included if the metadata field "is_custom_probes_used" is "Yes". The file should minimally include:target gene id, probe seq, probe id. The contents of this file are modeled after the 10x Genomics probe set file (see <https://support.10xgenomics.com/spatial-gene-expression-ffpe/probe-sets/probe-set-file-descriptions/probe-set-file-descriptions#probe_set_csv_file>).
+  -
+    pattern: raw\/custom_probe_set\.bed
+    required: False
+    description: This is a BED file version of the custom probe set file.
+  -
+    pattern: raw\/transcript_locations\.csv
+    required: True
+    description: "The origin of the coordinate is 0,0 at the top left corner of the image. The file should include: gene name, x, y, z (optional), quality score (optional). It is expected that the first row in the file contains the column header."
+  -
+    pattern: raw\/custom_gene_list\.csv
+    required: False
+    description: "This describes the target genes profiled by the assay. For advanced design, this can be probes sequences for splicing or other analysis for any target of interest. The format should minimally contain: gene name, ensemble ID"
+  -
+    pattern: raw\/probes\.csv
+    required: False
+    description: A CSV file describing the probe panel used. This is tyipcally what's used to specific the probe set when ordering a probe panel for a Xenium run.
+  -
+    pattern: raw\/gene_panel\.json
+    required: True
+    description: This is the JSON file describing the probes, as output from the xenium-ranger pipeline.
+  -
+    pattern: raw\/images\/overlay\.(?:jpeg|tiff)
+    required: False
+    description: State whether an overlay image was used to guide ROI selection. If an overlay is used, then the overlay details will be provided in the protocols.io protocol. If used, this needs to be uploaded. It is not included in the OME TIFF. This can be a JPEG or TIFF file
+  -
+    pattern: lab_processed\/.*
+    required: True
+    description: Experiment files that were processed by the lab generating the data.
+  -
+    pattern: lab_processed\/images\/.*
+    required: True
+    description: Processed image files
+  -
+    pattern: lab_processed\/images\/[^\/]+\.ome\.tiff
+    required: True
+    description: OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. For Visium this stitched file should only include the single capture area relevant to the current dataset. For GeoMx there will be one OME TIFF file per slide, with each slide including multiple AOIs. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0>
+    is_qa_qc: False
+    example: lab_processed/images/HBM892.MDXS.293.ome.tiff
+  -
+    pattern: lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv
+    required: True
+    description: This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0>
+    is_qa_qc: False