-
Notifications
You must be signed in to change notification settings - Fork 18
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Juranic/segmasks dirschema update (#1350)
* Create segmentation-mask-v2.1.yaml Update segmentation mask directory schema * Update CHANGELOG.md * Update segmentation-mask-v2.0.yaml Moved from v2.1 to v2.0 since segmentation masks were never implemented as a standalone assay. * Delete src/ingest_validation_tools/directory-schemas/segmentation-mask-v2.1.yaml Deleted, file was moved to v2.0 * Update imc-2d-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update codex-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update af-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update confocal-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update enhanced-srs-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update lightsheet-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update second-harmonic-generation-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update thick-section-multiphoton-mxif-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update visium-with-probes-v2.1.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * Update histology-v2.0.yaml Remove segmentation masks (lab_processed\/annotations\/.*, that has segmentation-masks-v2 dependency) * General: Remove "dependencies" functionality * General: Update seg mask assay yaml * General: Update seg mask assay docs * General: Syntax fix * General: Syntax fix --------- Co-authored-by: jpuerto-psc <[email protected]> Co-authored-by: Juan Puerto <=>
- Loading branch information
1 parent
31f3cf3
commit f23c3db
Showing
31 changed files
with
331 additions
and
566 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,34 +30,26 @@ This schema is for autofluorescence (AF). For an example of an AF dataset & dire | |
| ## Directory schemas | ||
| <summary><b>Version 2.0 (use this one)</b></summary> | ||
|
|
||
| | pattern | required? | description | dependent on | | ||
| | --- | --- | --- | --- | | ||
| | <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] | | | ||
| | <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | | ||
| | <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | | ||
| | <code>raw\/.*</code> | ✓ | Raw data files for the experiment. | | | ||
| | <code>raw\/channel_layout\.tsv</code> | ✓ | Table that includes a dictionary for channel to moiety, which may be a protein given in an OMAP panel or captured in the ASCT+B table. | | | ||
| | <code>raw\/images\/.*</code> | ✓ | Raw image files. Using this subdirectory allows for harmonization with other imaging assays. [This directory must include at least one raw file.] | | | ||
| | <code>raw\/images\/[^\/]+\.(?:xml|nd2|oir|lif|czi|tiff)</code> | ✓ | Raw microscope file for the experiment | | | ||
| | <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. | | | ||
| | <code>lab_processed\/images\/.*</code> | ✓ | Processed image files | | | ||
| | <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> | | | ||
| | <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> | | | ||
| | <code>lab_processed\/transformations\/.*</code> | | This directory contains transformation matrices that capture how each modality is aligned with the other and can be used to visualize overlays of multimodal data. This is needed to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). In these cases data type may have different pixel sizes and slightly different orientations (i.e., one may be rotated relative to another). | | | ||
| | <code>lab_processed\/transformations\/[^\/]+\.txt</code> | | Transformation matrices used to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). | | | ||
| | <code>qa_qc\/.*</code> | ✓ | Directory containing QA and/or QC information. | | | ||
| | <code>qa_qc\/resolution_report\/.*</code> | ✓ | Directory containing the results of resolution tests and/or vendor preventative maintenance reports. | | | ||
| | <code>qa_qc\/resolution_report\/resolution\.txt</code> | | This file summarizes the results of resolution tests or vendor reports from preventative maintenance visits. | | | ||
| | <code>qa_qc\/resolution_report\/[^\/]+\.pdf</code> | | This file is a pdf from a vendor preventative maintenance visit or resolution check tool demonstrating resolution. This file may include illumination test results. | | | ||
| | <code>qa_qc\/illumination_report\/.*</code> | ✓ | Directory containing the results of illumination tests and/or vendor preventative maintenance reports. | | | ||
| | <code>qa_qc\/illumination_report\/illumination.txt</code> | | This file summarizes the results of illumination tests or vendor reports from preventative maintenance visits. | | | ||
| | <code>qa_qc\/illumination_report\/[^\/]+\.pdf</code> | | This file is a pdf from a vendor preventative maintenance visit or illumination check tool demonstrating illumination intensity. | | | ||
| | <code>lab_processed\/annotations\/.*</code> | | Directory containing segmentation masks. | | | ||
| | <code>lab_processed\/annotations\/[^\/]+\.segmentations\.ome\.tiff</code> | | The segmentation masks should be stored as multi-channel pyramidal OME TIFF bitmasks with one channel per mask, where a single mask contains all instances of a type of object (e.g., all cells, a class of FTUs, etc). The class of objects contained in the mask is documented in the segmentation-masks.csv file. Each individual object in a mask should be represented by a unique integer pixel value starting at 1, with 0 meaning background (e.g., all pixels belonging to the first instance of a T-cell have a value of 1, the pixels for the second instance of a T-cell have a value of 2, etc). The pixel values should be unique within a mask. FTUs and other structural elements should be captured the same way as cells with segmentation masks and the appropriate channel feature definitions. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/segmentation-masks\.csv</code> | | This file contains details about each mask, with one row per mask. Each column in this file contains details describing the mask (e.g., channel number, mask name, ontological ID, etc). Each mask is stored as a channel in the segmentations.ome.tiff file and the mask name should be ontologically based and linked to the ASCT+B table where possible. The number of rows in this file should equal the number of channels in the segmentations.ome.tiff. For example, one row in this file would ontologically describe cells, if the segmentations.ome.tiff file contained a mask of all cells. A minimum set of fields (required and optional) is included below. If multiple segmentations.ome.tiff files are used, this segmentation-masks.csv file should document the masks across all of the OME TIFF files. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/[^\/]+-objects\.csv</code> | | This is a matrix where each row describes an individual object (e.g., one row per cell in the case where a mask contains all cells) and columns are features (i.e., object type, marker intensity, classification strategies, etc). One file should be created per mask with the name of the mask prepended to the file name. For example, if there’s a cell segmentation map called “cells” then you would include a file called “cells-objects.csv” and that file would contain one row per cell in the “cells” mask and one column per feature, such as marker intensity and/or cell type. A minimum set of fields (required and optional) is included below. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/[^\/]+\.geojson</code> | | A GeoJSON file(s) containing the geometries of each object within a mask. For example, if the mask contains multiple FTUs, multiple cells, etc, each of the objects in the mask would be independently documented in the GeoJSON file. There would be a single GeoJSON file per mask and the name of the file should be the name of the mask. If this file is generated by QuPath, the coordinates will be in pixel units with the origin (0, 0) as the top left corner of the full-resolution image. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/tissue-boundary\.geojson</code> | | **[QA/QC]** If the boundaries of the tissue have been identified (e.g., by manual efforts), then the boundary geometry can be included as a GeoJSON file named “tissue-boundary.geojson”. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/regions-of-concern\.csv</code> | | This file and the associated GeoJSON file can be used to denote any regions in the image that may contain QA/QC concerns. For example, if there are folds in the tissue, the region of the fold can be highlighted. This file should contain one row per region and include documentation about the region and why it's being flagged. | lab_processed\/annotations\/.* | | ||
| | <code>lab_processed\/annotations\/regions-of-concern\.geojson</code> | | This file and the associated CSV file can be used to denote any regions in the image that may contain QA/QC concerns. For example, if there are folds in the tissue, the region of the fold can be highlighted. This file should contain the geometric coordinates of each region being flagged. | lab_processed\/annotations\/.* | | ||
| | pattern | required? | description | | ||
| | --- | --- | --- | | ||
| | <code>extras\/.*</code> | ✓ | Folder for general lab-specific files related to the dataset. [Exists in all assays] | | ||
| | <code>extras\/microscope_hardware\.json</code> | ✓ | **[QA/QC]** A file generated by the micro-meta app that contains a description of the hardware components of the microscope. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | ||
| | <code>extras\/microscope_settings\.json</code> | | **[QA/QC]** A file generated by the micro-meta app that contains a description of the settings that were used to acquire the image data. Email HuBMAP Consortium Help Desk <[email protected]> if help is required in generating this document. | | ||
| | <code>raw\/.*</code> | ✓ | Raw data files for the experiment. | | ||
| | <code>raw\/channel_layout\.tsv</code> | ✓ | Table that includes a dictionary for channel to moiety, which may be a protein given in an OMAP panel or captured in the ASCT+B table. | | ||
| | <code>raw\/images\/.*</code> | ✓ | Raw image files. Using this subdirectory allows for harmonization with other imaging assays. [This directory must include at least one raw file.] | | ||
| | <code>raw\/images\/[^\/]+\.(?:xml|nd2|oir|lif|czi|tiff)</code> | ✓ | Raw microscope file for the experiment | | ||
| | <code>lab_processed\/.*</code> | ✓ | Experiment files that were processed by the lab generating the data. | | ||
| | <code>lab_processed\/images\/.*</code> | ✓ | Processed image files | | ||
| | <code>lab_processed\/images\/[^\/]+\.ome\.tiff</code> (example: <code>lab_processed/images/HBM892.MDXS.293.ome.tiff</code>) | ✓ | OME-TIFF files (multichannel, multi-layered) produced by the microscopy experiment. If compressed, must use loss-less compression algorithm. See the following link for the set of fields that are required in the OME TIFF file XML header. <https://docs.google.com/spreadsheets/d/1YnmdTAA0Z9MKN3OjR3Sca8pz-LNQll91wdQoRPSP6Q4/edit#gid=0> | | ||
| | <code>lab_processed\/images\/[^\/]*ome-tiff\.channels\.csv</code> | ✓ | This file provides essential documentation pertaining to each channel of the accommpanying OME TIFF. The file should contain one row per OME TIFF channel. The required fields are detailed <https://docs.google.com/spreadsheets/d/1xEJSb0xn5C5fB3k62pj1CyHNybpt4-YtvUs5SUMS44o/edit#gid=0> | | ||
| | <code>lab_processed\/transformations\/.*</code> | | This directory contains transformation matrices that capture how each modality is aligned with the other and can be used to visualize overlays of multimodal data. This is needed to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). In these cases data type may have different pixel sizes and slightly different orientations (i.e., one may be rotated relative to another). | | ||
| | <code>lab_processed\/transformations\/[^\/]+\.txt</code> | | Transformation matrices used to overlay images from the exact same tissue section (e.g., MALDI imaging mass spec, autofluorescence microscopy, MxIF, histological stains). | | ||
| | <code>qa_qc\/.*</code> | ✓ | Directory containing QA and/or QC information. | | ||
| | <code>qa_qc\/resolution_report\/.*</code> | ✓ | Directory containing the results of resolution tests and/or vendor preventative maintenance reports. | | ||
| | <code>qa_qc\/resolution_report\/resolution\.txt</code> | | This file summarizes the results of resolution tests or vendor reports from preventative maintenance visits. | | ||
| | <code>qa_qc\/resolution_report\/[^\/]+\.pdf</code> | | This file is a pdf from a vendor preventative maintenance visit or resolution check tool demonstrating resolution. This file may include illumination test results. | | ||
| | <code>qa_qc\/illumination_report\/.*</code> | ✓ | Directory containing the results of illumination tests and/or vendor preventative maintenance reports. | | ||
| | <code>qa_qc\/illumination_report\/illumination.txt</code> | | This file summarizes the results of illumination tests or vendor reports from preventative maintenance visits. | | ||
| | <code>qa_qc\/illumination_report\/[^\/]+\.pdf</code> | | This file is a pdf from a vendor preventative maintenance visit or illumination check tool demonstrating illumination intensity. | | ||
|
|
||
Oops, something went wrong.