DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation

• URL: http://arxiv.org/abs/2412.12492v1
• Date: Tue, 17 Dec 2024 02:47:00 GMT
• Title: DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation
• Authors: Qingtao Pan, Wenhao Qiao, Jingjiao Lou, Bing Ji, Shuo Li,
• Abstract summary: Semi-supervised medical image segmentation (SSMIS) uses consistency learning to regularize model training. SSMIS often suffers from error supervision from low-quality pseudo labels. We propose a Dual Semantic Similarity-Supervised VLM (DuSSS) for SSMIS.
• Score: 4.523111195300109
• License:
• Abstract: Semi-supervised medical image segmentation (SSMIS) uses consistency learning to regularize model training, which alleviates the burden of pixel-wise manual annotations. However, it often suffers from error supervision from low-quality pseudo labels. Vision-Language Model (VLM) has great potential to enhance pseudo labels by introducing text prompt guided multimodal supervision information. It nevertheless faces the cross-modal problem: the obtained messages tend to correspond to multiple targets. To address aforementioned problems, we propose a Dual Semantic Similarity-Supervised VLM (DuSSS) for SSMIS. Specifically, 1) a Dual Contrastive Learning (DCL) is designed to improve cross-modal semantic consistency by capturing intrinsic representations within each modality and semantic correlations across modalities. 2) To encourage the learning of multiple semantic correspondences, a Semantic Similarity-Supervision strategy (SSS) is proposed and injected into each contrastive learning process in DCL, supervising semantic similarity via the distribution-based uncertainty levels. Furthermore, a novel VLM-based SSMIS network is designed to compensate for the quality deficiencies of pseudo-labels. It utilizes the pretrained VLM to generate text prompt guided supervision information, refining the pseudo label for better consistency regularization. Experimental results demonstrate that our DuSSS achieves outstanding performance with Dice of 82.52%, 74.61% and 78.03% on three public datasets (QaTa-COV19, BM-Seg and MoNuSeg).

Related papers

• Estimating Commonsense Plausibility through Semantic Shifts [66.06254418551737]
  We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts. Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
  arXiv  Detail & Related papers  (2025-02-19T06:31:06Z)
• Multi-Memory Matching for Unsupervised Visible-Infrared Person Re-Identification [30.983346937558743]
  Key challenges in USL-VI-ReID are to effectively generate pseudo-labels and establish pseudo-label correspondences. We propose a Multi-Memory Matching framework for USL-VI-ReID. Experiments on the public SYSU-MM01 and RegDB datasets demonstrate the reliability of the established cross-modality correspondences.
  arXiv  Detail & Related papers  (2024-01-12T01:24:04Z)
• SMC-NCA: Semantic-guided Multi-level Contrast for Semi-supervised Temporal Action Segmentation [53.010417880335424]
  Semi-supervised temporal action segmentation (SS-TA) aims to perform frame-wise classification in long untrimmed videos. Recent studies have shown the potential of contrastive learning in unsupervised representation learning using unlabelled data. We propose a novel Semantic-guided Multi-level Contrast scheme with a Neighbourhood-Consistency-Aware unit (SMC-NCA) to extract strong frame-wise representations.
  arXiv  Detail & Related papers  (2023-12-19T17:26:44Z)
• Cross-head mutual Mean-Teaching for semi-supervised medical image segmentation [6.738522094694818]
  Semi-supervised medical image segmentation (SSMIS) has witnessed substantial advancements by leveraging limited labeled data and abundant unlabeled data. Existing state-of-the-art (SOTA) methods encounter challenges in accurately predicting labels for the unlabeled data. We propose a novel Cross-head mutual mean-teaching Network (CMMT-Net) incorporated strong-weak data augmentation.
  arXiv  Detail & Related papers  (2023-10-08T09:13:04Z)
• DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning [37.48292304239107]
  We present a transformer-based end-to-end ZSL method named DUET. We develop a cross-modal semantic grounding network to investigate the model's capability of disentangling semantic attributes from the images. We find that DUET can often achieve state-of-the-art performance, its components are effective and its predictions are interpretable.
  arXiv  Detail & Related papers  (2022-07-04T11:12:12Z)
• Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentation [119.009033745244]
  This paper presents a Self-supervised Low-Rank Network ( SLRNet) for single-stage weakly supervised semantic segmentation (WSSS) and semi-supervised semantic segmentation (SSSS) SLRNet uses cross-view self-supervision, that is, it simultaneously predicts several attentive LR representations from different views of an image to learn precise pseudo-labels. Experiments on the Pascal VOC 2012, COCO, and L2ID datasets demonstrate that our SLRNet outperforms both state-of-the-art WSSS and SSSS methods with a variety of different settings.
  arXiv  Detail & Related papers  (2022-03-19T09:19:55Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Information Symmetry Matters: A Modal-Alternating Propagation Network for Few-Shot Learning [118.45388912229494]
  We propose a Modal-Alternating Propagation Network (MAP-Net) to supplement the absent semantic information of unlabeled samples. We design a Relation Guidance (RG) strategy to guide the visual relation vectors via semantics so that the propagated information is more beneficial. Our proposed method achieves promising performance and outperforms the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-09-03T03:43:53Z)
• Learning Relation Alignment for Calibrated Cross-modal Retrieval [52.760541762871505]
  We propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. We present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions mutually via inter-modal alignment.
  arXiv  Detail & Related papers  (2021-05-28T14:25:49Z)
• Weakly supervised segmentation with cross-modality equivariant constraints [7.757293476741071]
  Weakly supervised learning has emerged as an appealing alternative to alleviate the need for large labeled datasets in semantic segmentation. We present a novel learning strategy that leverages self-supervision in a multi-modal image scenario to significantly enhance original CAMs. Our approach outperforms relevant recent literature under the same learning conditions.
  arXiv  Detail & Related papers  (2021-04-06T13:14:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.