Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis

• URL: http://arxiv.org/abs/2501.03565v1
• Date: Tue, 07 Jan 2025 06:30:52 GMT
• Title: Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis
• Authors: Haoran Lai, Zihang Jiang, Qingsong Yao, Rongsheng Wang, Zhiyang He, Xiaodong Tao, Wei Wei, Weifu Lv, S. Kevin Zhou,
• Abstract summary: 3D medical images such as Computed tomography (CT) are widely used in clinical practice, offering a great potential for automatic diagnosis. Supervised learning-based approaches have achieved significant progress but rely heavily on extensive manual annotations. Vision-language alignment (VLA) offers a promising alternative by enabling zero-shot learning without additional annotations.
• Score: 23.56751925900571
• License:
• Abstract: 3D medical images such as Computed tomography (CT) are widely used in clinical practice, offering a great potential for automatic diagnosis. Supervised learning-based approaches have achieved significant progress but rely heavily on extensive manual annotations, limited by the availability of training data and the diversity of abnormality types. Vision-language alignment (VLA) offers a promising alternative by enabling zero-shot learning without additional annotations. However, we empirically discover that the visual and textural embeddings after alignment endeavors from existing VLA methods form two well-separated clusters, presenting a wide gap to be bridged. To bridge this gap, we propose a Bridged Semantic Alignment (BrgSA) framework. First, we utilize a large language model to perform semantic summarization of reports, extracting high-level semantic information. Second, we design a Cross-Modal Knowledge Interaction (CMKI) module that leverages a cross-modal knowledge bank as a semantic bridge, facilitating interaction between the two modalities, narrowing the gap, and improving their alignment. To comprehensively evaluate our method, we construct a benchmark dataset that includes 15 underrepresented abnormalities as well as utilize two existing benchmark datasets. Experimental results demonstrate that BrgSA achieves state-of-the-art performances on both public benchmark datasets and our custom-labeled dataset, with significant improvements in zero-shot diagnosis of underrepresented abnormalities.

Related papers

• A Learnable Multi-views Contrastive Framework with Reconstruction Discrepancy for Medical Time-Series [8.741139851597364]
  We propose incorporating external data from related tasks and leveraging AE-GAN to extract prior knowledge. We introduce LMCF, a framework that integrates a multi-head attention mechanism and adaptively learns representations from different views. Experiments on three target datasets demonstrate that our method consistently outperforms seven other baselines.
  arXiv  Detail & Related papers  (2025-01-30T14:20:11Z)
• A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation [12.948027961485536]
  We propose a novel Weakly Supervised Semantic (WSSS) approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels. Our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.
  arXiv  Detail & Related papers  (2024-11-19T16:20:27Z)
• Manifold-Aware Local Feature Modeling for Semi-Supervised Medical Image Segmentation [20.69908466577971]
  We introduce the Manifold-Aware Local Feature Modeling Network (MANet), which enhances the U-Net architecture by incorporating manifold supervision signals. Our experiments on datasets such as ACDC, LA, and Pancreas-NIH demonstrate that MANet consistently surpasses state-of-the-art methods in performance metrics.
  arXiv  Detail & Related papers  (2024-10-14T08:40:35Z)
• DuEDL: Dual-Branch Evidential Deep Learning for Scribble-Supervised Medical Image Segmentation [2.708515419272247]
  We propose a novel framework called Dual-Branch Evi-dential Deep Learning (DuEDL) Our method significantly enhances the reliability and generalization ability of the model without sacrificing accuracy, outper-forming state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-05-23T11:23:57Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• Exploring Weakly Supervised Semantic Segmentation Ensembles for Medical Imaging Systems [11.693197342734152]
  We propose a framework for reliable classification and detection of medical conditions in images. Our framework achieves that by first utilizing lower threshold CAMs to cover the target object with high certainty. We have demonstrated an improved dice score of up to 8% on BRATS and 6% on DECATHLON datasets.
  arXiv  Detail & Related papers  (2023-03-14T13:31:05Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Learning Binary Semantic Embedding for Histology Image Classification and Retrieval [56.34863511025423]
  We propose a novel method for Learning Binary Semantic Embedding (LBSE) Based on the efficient and effective embedding, classification and retrieval are performed to provide interpretable computer-assisted diagnosis for histology images. Experiments conducted on three benchmark datasets validate the superiority of LBSE under various scenarios.
  arXiv  Detail & Related papers  (2020-10-07T08:36:44Z)
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis [102.40869566439514]
  We seek to exploit rich labeled data from relevant domains to help the learning in the target task via Unsupervised Domain Adaptation (UDA) Unlike most UDA methods that rely on clean labeled data or assume samples are equally transferable, we innovatively propose a Collaborative Unsupervised Domain Adaptation algorithm. We theoretically analyze the generalization performance of the proposed method, and also empirically evaluate it on both medical and general images.
  arXiv  Detail & Related papers  (2020-07-05T11:49:17Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.