Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models

• URL: http://arxiv.org/abs/2508.18886v1
• Date: Tue, 26 Aug 2025 10:01:23 GMT
• Title: Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models
• Abstract summary: Vision-language models (VLMs) exhibit strong generalization, and text prompts encode identity attributes.<n>We propose DualFairVL, a multimodal prompt-learning framework that jointly debiases and aligns cross-modal representations.<n>Experiments show that DualFairVL achieves state-of-the-art fairness and accuracy under both in- and out-of-distribution settings.
• Score: 34.42749153279352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensuring fairness across demographic groups in medical diagnosis is essential for equitable healthcare, particularly under distribution shifts caused by variations in imaging equipment and clinical practice. Vision-language models (VLMs) exhibit strong generalization, and text prompts encode identity attributes, enabling explicit identification and removal of sensitive directions. However, existing debiasing approaches typically address vision and text modalities independently, leaving residual cross-modal misalignment and fairness gaps. To address this challenge, we propose DualFairVL, a multimodal prompt-learning framework that jointly debiases and aligns cross-modal representations. DualFairVL employs a parallel dual-branch architecture that separates sensitive and target attributes, enabling disentangled yet aligned representations across modalities. Approximately orthogonal text anchors are constructed via linear projections, guiding cross-attention mechanisms to produce fused features. A hypernetwork further disentangles attribute-related information and generates instance-aware visual prompts, which encode dual-modal cues for fairness and robustness. Prototype-based regularization is applied in the visual branch to enforce separation of sensitive features and strengthen alignment with textual anchors. Extensive experiments on eight medical imaging datasets across four modalities show that DualFairVL achieves state-of-the-art fairness and accuracy under both in- and out-of-distribution settings, outperforming full fine-tuning and parameter-efficient baselines with only 3.6M trainable parameters. Code will be released upon publication.

Related papers

• DistMedVL: Distributional Vision-Language Alignment for Uncertainty-Aware Medical Image Segmentation [16.13084489571617]
  Cross-modal alignment of visual and textual representations is fundamental to multimodal medical image understanding.<n>Existing vision-language segmentation methods rely on deterministic cross-modal matching, which overlooks aleatoric uncertainty from ambiguous boundaries.<n>We propose DistMedVL, a probabilistic vision-language framework that introduces a lightweight Probabilistic Cross-Modal Adapter (PCM-Adapter) upon frozen encoders to explicitly model representational uncertainty.
  arXiv  Detail & Related papers  (2026-08-06T07:17:14Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptation [13.030685683550317]
  Biomedical Vision--Language Models (VLMs) have shown remarkable promise in few-shot medical diagnosis.<n>Existing frameworks typically optimize visual and textual prompts as independent streams.<n>BiomedAP is a vision-informed dual-anchor framework with gated cross-modal fusion.
  arXiv  Detail & Related papers  (2026-05-15T08:45:57Z)
• Cite-While-You-Generate: Training-Free Evidence Attribution for Multimodal Clinical Summarization [32.47484883374212]
  Trustworthy clinical summarization requires fluent generation and transparency about where each statement comes from.<n>We propose a training-free framework for generation-time source attribution that leverages decoder attentions to directly cite supporting text spans or images.<n>We introduce two strategies for multimodal attribution: a raw image mode, which directly uses image patch attentions, and a caption-as-span mode, which substitutes images with generated captions to enable purely text-based alignment.
  arXiv  Detail & Related papers  (2026-01-23T02:01:43Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image.<n> Vision-Language Pre-training models offer a strong open-vocabulary foundation, but struggle with fine-grained localization under weak supervision.<n>We propose the Dual Adaptive Refinement Transfer (DART) framework to overcome these limitations.
  arXiv  Detail & Related papers  (2025-08-07T17:22:33Z)
• On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI [4.866086225040713]
  We introduce a perturbation-based approach to quantify a model's reliance on each modality in binary classification tasks.<n>By swapping images or text between samples with opposing labels, we expose modality-specific biases.
  arXiv  Detail & Related papers  (2025-07-31T21:35:52Z)
• Dual-branch Prompting for Multimodal Machine Translation [9.903997553625253]
  We propose D2P-MMT, a diffusion-based dual-branch prompting framework for robust vision-guided translation.<n>D2P-MMT requires only the source text and a reconstructed image generated by a pre-trained diffusion model.<n>Experiments on the Multi30K dataset demonstrate that D2P-MMT achieves superior translation performance compared to existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-07-23T15:22:51Z)
• Beyond Quantity: Distribution-Aware Labeling for Visual Grounding [72.43984105242177]
  Visual grounding requires large and diverse region-text pairs.<n>Existing pseudo-labeling pipelines often overfit to biased distributions.<n>We propose DAL, a distribution-aware labeling framework for visual grounding.
  arXiv  Detail & Related papers  (2025-05-30T09:04:47Z)
• BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation [9.262045402495225]
  BiPVL-Seg is an end-to-end framework that integrates vision-language fusion and embedding alignment.<n>BiPVL-Seg introduces progressive fusion in the architecture, which facilitates stage-wise information exchange between vision and text encoders.<n>It incorporates global-local contrastive alignment, a training objective that enhances the text encoder's comprehension by aligning text and vision embeddings at both class and concept levels.
  arXiv  Detail & Related papers  (2025-03-30T17:34:39Z)
• CM-Diff: A Single Generative Network for Bidirectional Cross-Modality Translation Diffusion Model Between Infrared and Visible Images [48.57429642590462]
  We present the bidirectional cross-modality translation diffusion model (CM-Diff) for simultaneously modeling data distributions in both the infrared and visible modalities.<n> Experimental results demonstrate the superiority of our CM-Diff over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-12T16:25:18Z)
• Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis [23.56751925900571]
  3D medical images such as Computed tomography (CT) are widely used in clinical practice, offering a great potential for automatic diagnosis.<n>Supervised learning-based approaches have achieved significant progress but rely heavily on extensive manual annotations.<n> Vision-language alignment (VLA) offers a promising alternative by enabling zero-shot learning without additional annotations.
  arXiv  Detail & Related papers  (2025-01-07T06:30:52Z)
• DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation [4.523111195300109]
  Semi-supervised medical image segmentation (SSMIS) uses consistency learning to regularize model training.<n>SSMIS often suffers from error supervision from low-quality pseudo labels.<n>We propose a Dual Semantic Similarity-Supervised VLM (DuSSS) for SSMIS.
  arXiv  Detail & Related papers  (2024-12-17T02:47:00Z)
• PV2TEA: Patching Visual Modality to Textual-Established Information Extraction [59.76117533540496]
  We patch the visual modality to the textual-established attribute information extractor. PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
  arXiv  Detail & Related papers  (2023-06-01T05:39:45Z)
• Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation [48.723504098917324]
  We propose an Unify, Align and then Refine (UAR) approach to learn multi-level cross-modal alignments. We introduce three novel modules: Latent Space Unifier, Cross-modal Representation Aligner and Text-to-Image Refiner. Experiments and analyses on IU-Xray and MIMIC-CXR benchmark datasets demonstrate the superiority of our UAR against varied state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-28T12:42:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.