Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading

• URL: http://arxiv.org/abs/2408.08527v1
• Date: Fri, 16 Aug 2024 04:54:10 GMT
• Title: Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading
• Authors: Li Pan, Yupei Zhang, Qiushi Yang, Tan Li, Xiaohan Xing, Maximus C. F. Yeung, Zhen Chen,
• Abstract summary: multimodal deep learning has achieved a promising performance in glioma grading. We introduce a novel Focus on Focus (FoF) framework with paired pathology-genomic training and applicable pathology-only inference. Experiments on the TCGA GBM-LGG dataset demonstrate that our FoF framework significantly improves the glioma grading.
• Score: 15.492493684721175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two issues hinder existing methods to precisely interpret diagnostic molecular-pathology features, thereby limiting their grading performance. Moreover, the real-world applicability of existing multimodal approaches is significantly restricted as molecular biomarkers are not always available during clinical deployment. To address these problems, we introduce a novel Focus on Focus (FoF) framework with paired pathology-genomic training and applicable pathology-only inference, enhancing molecular-pathology representation effectively. Specifically, we propose a Focus-oriented Representation Learning (FRL) module to encourage the model to identify regions positively or negatively related to glioma grading and guide it to focus on the diagnostic areas with a consistency constraint. To effectively link the molecular biomarkers to morphological features, we propose a Multi-view Cross-modal Alignment (MCA) module that projects histopathology representations into molecular subspaces, aligning morphological features with corresponding molecular biomarker status by supervised contrastive learning. Experiments on the TCGA GBM-LGG dataset demonstrate that our FoF framework significantly improves the glioma grading. Remarkably, our FoF achieves superior performance using only histopathology slides compared to existing multimodal methods. The source code is available at https://github.com/peterlipan/FoF.

Related papers

• FARM: Functional Group-Aware Representations for Small Molecules [55.281754551202326]
  We introduce Functional Group-Aware Representations for Small Molecules (FARM) FARM is a foundation model designed to bridge the gap between SMILES, natural language, and molecular graphs. We rigorously evaluate FARM on the MoleculeNet dataset, where it achieves state-of-the-art performance on 10 out of 12 tasks.
  arXiv  Detail & Related papers  (2024-10-02T23:04:58Z)
• GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation [68.63955715643974]
  Modality-prompted Heterogeneous Graph for Omnimodal Learning (GTP-4o) We propose an innovative Modality-prompted Heterogeneous Graph for Omnimodal Learning (GTP-4o)
  arXiv  Detail & Related papers  (2024-07-08T01:06:13Z)
• ProBio: A Protocol-guided Multimodal Dataset for Molecular Biology Lab [67.24684071577211]
  The challenge of replicating research results has posed a significant impediment to the field of molecular biology. We first curate a comprehensive multimodal dataset, named ProBio, as an initial step towards this objective. Next, we devise two challenging benchmarks, transparent solution tracking and multimodal action recognition, to emphasize the unique characteristics and difficulties associated with activity understanding in BioLab settings.
  arXiv  Detail & Related papers  (2023-11-01T14:44:01Z)
• Deep Learning Predicts Biomarker Status and Discovers Related Histomorphology Characteristics for Low-Grade Glioma [21.281553456323998]
  Biomarker detection is an indispensable part in the diagnosis and treatment of low-grade glioma (LGG) We propose an interpretable deep learning pipeline to predict the status of five biomarkers in LGG using only hematoxylin and eosin-stained whole slide images and slide-level biomarker status labels. Our pipeline not only provides a novel approach for biomarker prediction, enhancing the applicability of molecular treatments for LGG patients but also facilitates the discovery of new mechanisms in molecular functionality and LGG progression.
  arXiv  Detail & Related papers  (2023-10-11T13:05:33Z)
• MolFM: A Multimodal Molecular Foundation Model [9.934141536012596]
  MolFM is a multimodal molecular foundation model designed to facilitate joint representation learning from molecular structures, biomedical texts, and knowledge graphs. We provide theoretical analysis that our cross-modal pre-training captures local and global molecular knowledge by minimizing the distance in the feature space between different modalities of the same molecule. On cross-modal retrieval, MolFM outperforms existing models with 12.13% and 5.04% absolute gains under the zero-shot and fine-tuning settings, respectively.
  arXiv  Detail & Related papers  (2023-06-06T12:45:15Z)
• Democratizing Pathological Image Segmentation with Lay Annotators via Molecular-empowered Learning [20.11220024755348]
  We propose a molecular-empowered learning scheme for multi-class cell segmentation using partial labels from lay annotators. Our learning method achieved F1 = 0.8496 using molecular-informed annotations from lay annotators. Our method democratizes the development of a pathological segmentation deep model to the lay annotator level.
  arXiv  Detail & Related papers  (2023-05-31T16:54:47Z)
• Multi-task Learning of Histology and Molecular Markers for Classifying Diffuse Glioma [9.082753496844731]
  We propose a hierarchical multi-task multi-instance learning framework to jointly predict histology and molecular markers. We also propose a co-occurrence probability-based label correction graph network to model the co-occurrence of molecular markers. Our experiments show that our method outperforms other state-of-the-art methods in classifying diffuse glioma.
  arXiv  Detail & Related papers  (2023-03-26T23:00:00Z)
• Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging [59.79875531898648]
  DeepGlioma is an artificial-intelligence-based diagnostic screening system. DeepGlioma can predict the molecular alterations used by the World Health Organization to define the adult-type diffuse glioma taxonomy.
  arXiv  Detail & Related papers  (2023-03-23T18:50:18Z)
• A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language [63.60376252491507]
  We propose a molecular multimodal foundation model which is pretrained from molecular graphs and their semantically related textual data. We believe that our model would have a broad impact on AI-empowered fields across disciplines such as biology, chemistry, materials, environment, and medicine.
  arXiv  Detail & Related papers  (2022-09-12T00:56:57Z)
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding [60.89561661441736]
  We propose a novel method for few-shot medical image segmentation. We construct our few-shot image segmentor using a deep convolutional network trained episodically. We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
  arXiv  Detail & Related papers  (2020-12-10T04:01:07Z)
• Interpretable multimodal fusion networks reveal mechanisms of brain cognition [26.954460880062506]
  We develop an interpretable multimodal fusion model, gCAM-CCL, which can perform automated diagnosis and result interpretation simultaneously. We validate the gCAM-CCL model on a brain imaging-genetic study, and show gCAM-CCL's performed well for both classification and mechanism analysis.
  arXiv  Detail & Related papers  (2020-06-16T18:52:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.