[AAAI 2025] MEATRD: Multimodal Anomalous Tissue Region Detection Enhanced with Spatial Transcriptomics

About

The detection of anomalous tissue regions (ATRs) within affected tissues is crucial in clinical diagnosis and pathological studies. Conventional automated ATR detection methods, primarily based on histology images alone, falter in cases where ATRs and normal tissues have subtle visual differences. The recent spatial transcriptomics (ST) technology profiles gene expressions across tissue regions, offering a molecular perspective for detecting ATRs. However, there is a dearth of ATR detection methods that effectively harness complementary information from both histology images and ST. To address this gap, we propose MEATRD, a novel ATR detection method that integrates histology image and ST data. MEATRD is trained to reconstruct image patches and gene expression profiles of normal tissue spots (inliers) from their multimodal embeddings, followed by learning a one-class classification AD model based on latent multimodal reconstruction errors. This strategy harmonizes the strengths of reconstruction-based and one-class classification approaches. At the heart of MEATRD is an innovative masked graph dual-attention transformer (MGDAT) network, which not only facilitates cross-modality and cross-node information sharing but also addresses the model over-generalization issue commonly seen in reconstruction-based AD methods. Additionally, we demonstrate that modality-specific, task-relevant information is collated and condensed in multimodal bottleneck encoding generated in MGDAT, marking the first theoretical analysis of the informational properties of multimodal bottleneck encoding. Extensive evaluations across eight real ST datasets reveal MEATRD's superior performance in ATR detection, surpassing various state-of-the-art AD methods. Remarkably, MEATRD also proves adept at discerning ATRs that only show slight visual deviations from normal tissues.

Dataset

• The raw AnnData files of 10x-hNB can be downloaded from here
• The raw AnnData files of 10x-hBC-{A1-H1} can be downloaded from here
• The raw AnnData files of 10x-hBC-I1 can be downloaded from here
• The preprocessed dataset can be downloaded from here

Code organization

• configs/ - Configuration files for each task and dataset
• data/ - The dataset is pre-processed by DGL
• utils.py - Util functions (including evaluation suite)
• main.py - Main python script for running anomaly detection task
• pretrain.py - Python script for pretraining MobileUNet
• build_datasets.py - Python scripts for preprocessing
• MobileUNet.pth - The pre-trained weights for MobileUNet

In the folder model:

• model/fusion.py - Multi-modal fusion modules
• model/meatrd.py - MEATRD framework for step II and step III
• model/loss.py - Loss function of MEATRD
• model/unet.py - MobileUNet of MEATRD

Contact

If you have any questions, please contact [email protected].