DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation

• URL: http://arxiv.org/abs/2608.05683v1
• Date: Thu, 06 Aug 2026 07:17:14 GMT
• Title: DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation
• Abstract summary: Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding.<n>Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries.<n>We propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty.
• Score: 16.13084489571617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding, yet remains hindered by uncertainty in both modalities under real-world clinical conditions. Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries and epistemic uncertainty from limited training data, leading to fragile performance under domain shift. To address this issue, we propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty. Specifically, the PCM-Adapter comprises two sequential modules for progressive probabilistic alignment. We first devise a Mahalanobis Alignment Module (MAM) that models textual tokens as Gaussian distributions and computes patch-text compatibility via Mahalanobis distance, yielding variance-conditioned matching that downweights unreliable feature dimensions. Moreover, we devise a Distribution Flow Module (DFM) that estimates modality-wise confidence parameters and performs vision-guided refinement of textual distributions, accommodating distributional variation across imaging modalities. Extensive experiments across eight medical segmentation benchmarks demonstrate that DistMedVL outperforms state-of-the-art methods with only 6.3M trainable parameters, exhibiting superior data efficiency, perturbation robustness and cross-dataset generalization.

Related papers

• Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke [4.272151451487239]
  This paper proposes a novel tri-modal fusion model for ischemic stroke prognosis.<n>We employ a Large Language Model (LLM) to automatically generate semi-structured diagnostic text from brain MRIs.<n>We also design a core component termed the Vision-Conditioned Dual Alignment Fusion Module.
  arXiv  Detail & Related papers  (2026-05-14T11:28:57Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Delving Aleatoric Uncertainty in Medical Image Segmentation via Vision Foundation Models [56.29123284262618]
  This work proposes leveraging the universal representation capabilities of visual foundation models to estimate inherent data uncertainty.<n>We analyze the feature diversity of the model's decoded representations and quantify their singular value energy to define the semantic perception scale for each class.<n>Based on this foundation, we design two uncertainty-driven application strategies: (1) the aleatoric uncertainty-aware data filtering mechanism to eliminate potentially noisy samples and enhance model learning quality; and (2) the dynamic uncertainty-aware optimization strategy that adaptively adjusts class-specific loss weights during training based on the semantic perception scale, combined with a label denoising mechanism to improve training stability
  arXiv  Detail & Related papers  (2026-04-13T03:59:54Z)
• MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation [8.913012426353154]
  We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation.<n>Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens.
  arXiv  Detail & Related papers  (2026-02-23T23:46:05Z)
• Uncertainty-Aware Vision-Language Segmentation for Medical Imaging [12.545486211087791]
  We introduce a novel uncertainty-aware multimodal segmentation framework for medical diagnosis.<n>We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion.<n>Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks.
  arXiv  Detail & Related papers  (2026-02-16T06:27:51Z)
• DiA-gnostic VLVAE: Disentangled Alignment-Constrained Vision Language Variational AutoEncoder for Robust Radiology Reporting with Missing Modalities [3.5045368873011924]
  We propose the DiA-gnostic VLVAE, which achieves robust radiology reporting through Disentangled Alignment.<n>Our framework is designed to be resilient to missing modalities by disentangling shared and modality-specific features.<n>A compact LLaMA-X decoder then uses these disentangled representations to generate reports efficiently.
  arXiv  Detail & Related papers  (2025-11-08T11:08:27Z)
• Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models [34.42749153279352]
  Vision-language models (VLMs) exhibit strong generalization, and text prompts encode identity attributes.<n>We propose DualFairVL, a multimodal prompt-learning framework that jointly debiases and aligns cross-modal representations.<n>Experiments show that DualFairVL achieves state-of-the-art fairness and accuracy under both in- and out-of-distribution settings.
  arXiv  Detail & Related papers  (2025-08-26T10:01:23Z)
• FlowSDF: Flow Matching for Medical Image Segmentation Using Distance Transforms [60.195642571004804]
  We introduce FlowSDF, an image-guided conditional flow matching framework, to represent an implicit distribution of segmentation masks.<n>Our framework enables accurate sampling of segmentation masks and the computation of relevant statistical measures.
  arXiv  Detail & Related papers  (2024-05-28T11:47:12Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)
• Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training [55.56609500764344]
  We propose a unified framework based on Multi-task Paired Masking with Alignment (MPMA) to integrate the cross-modal alignment task into the joint image-text reconstruction framework. We also introduce a Memory-Augmented Cross-Modal Fusion (MA-CMF) module to fully integrate visual information to assist report reconstruction.
  arXiv  Detail & Related papers  (2023-05-13T13:53:48Z)
• Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency [24.78258331561847]
  This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired medical images. We leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios.
  arXiv  Detail & Related papers  (2022-06-21T17:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.