Robust Fairness Vision-Language Learning for Medical Image Analysis

• URL: http://arxiv.org/abs/2505.03153v1
• Date: Tue, 06 May 2025 03:59:25 GMT
• Title: Robust Fairness Vision-Language Learning for Medical Image Analysis
• Authors: Sparsh Bansal, Mingyang Wu, Xin Wang, Shu Hu,
• Abstract summary: We introduce a framework for ensuring robustness and fairness of Vision-Language Models.<n>This framework modifies the loss function at training by identifying and adjusting faulty image-text pairs.<n>We show up to a 8.6% improvement when looking at equity-scaled AUC.
• Score: 11.848018894413556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The advent of Vision-Language Models (VLMs) in medical image analysis has the potential to help process multimodal inputs and increase performance over traditional inference methods. However, when considering the domain in which these models will be implemented, fairness and robustness are important to ensure the model stays true for any patient. In this paper, we introduce a framework for ensuring robustness and fairness of VLM models. This framework modifies the loss function at training by identifying and adjusting faulty image-text pairs through a Dynamic Bad Pair Mining algorithm and also utilizing Sinkhorn distance to ensure the loss distributions of protected groups do not deviate from the total loss. Experimental testing of our framework shows up to a 8.6\% improvement when looking at equity-scaled AUC.

Related papers

• On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable? [0.9626666671366837]
  We introduce MediMeta-C, a corruption benchmark that applies several perturbations across multiple medical imaging datasets.<n>We propose RobustMedCLIP, a visual encoder adaptation of a pretrained MVLM that incorporates few-shot tuning to enhance resilience against corruptions.
  arXiv  Detail & Related papers  (2025-05-21T12:08:31Z)
• Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation [26.580361841501514]
  Vision-language models (VLMs) excel in various multimodal tasks but frequently suffer from poor calibration.<n>This miscalibration undermines user trust, especially when models confidently provide incorrect or fabricated information.<n>We propose a novel Confidence through Semantic Perturbation (CSP) framework to improve the calibration of verbalized confidence for object-centric queries.
  arXiv  Detail & Related papers  (2025-04-21T04:01:22Z)
• Analyzing the Effect of $k$-Space Features in MRI Classification Models [0.0]
  We have developed an explainable AI methodology tailored for medical imaging. We employ a Convolutional Neural Network (CNN) that analyzes MRI scans across both image and frequency domains. This approach not only enhances early training efficiency but also deepens our understanding of how additional features impact the model predictions.
  arXiv  Detail & Related papers  (2024-09-20T15:43:26Z)
• Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving [55.93813178692077]
  We present RoboBEV, an extensive benchmark suite designed to evaluate the resilience of BEV algorithms.<n>We assess 33 state-of-the-art BEV-based perception models spanning tasks like detection, map segmentation, depth estimation, and occupancy prediction.<n>Our experimental results also underline the efficacy of strategies like pre-training and depth-free BEV transformations in enhancing robustness against out-of-distribution data.
  arXiv  Detail & Related papers  (2024-05-27T17:59:39Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study [61.65123150513683]
  multimodal foundation models, such as CLIP, produce state-of-the-art zero-shot results. It is reported that these models close the robustness gap by matching the performance of supervised models trained on ImageNet. We show that CLIP leads to a significant robustness drop compared to supervised ImageNet models on our benchmark.
  arXiv  Detail & Related papers  (2024-03-15T17:33:49Z)
• PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation [50.556961575275345]
  We propose a perception-aware fusion framework to promote segmentation robustness in adversarial scenes. We show that our scheme substantially enhances the robustness, with gains of 15.3% mIOU, compared with advanced competitors.
  arXiv  Detail & Related papers  (2023-08-08T01:55:44Z)
• Fairness Increases Adversarial Vulnerability [50.90773979394264]
  This paper shows the existence of a dichotomy between fairness and robustness, and analyzes when achieving fairness decreases the model robustness to adversarial samples. Experiments on non-linear models and different architectures validate the theoretical findings in multiple vision domains. The paper proposes a simple, yet effective, solution to construct models achieving good tradeoffs between fairness and robustness.
  arXiv  Detail & Related papers  (2022-11-21T19:55:35Z)
• Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications [72.52228843498193]
  In training deep learning models, a compromise often must be made between performance and trust. In this work, we integrate a new surrogate loss with self-supervised learning for computer-aided screening of COVID-19 patients.
  arXiv  Detail & Related papers  (2021-12-14T21:16:52Z)
• Robustness in Deep Learning for Computer Vision: Mind the gap? [13.576376492050185]
  We identify, analyze, and summarize current definitions and progress towards non-adversarial robustness in deep learning for computer vision. We find that this area of research has received disproportionately little attention relative to adversarial machine learning.
  arXiv  Detail & Related papers  (2021-12-01T16:42:38Z)
• Uncertainty-aware Generalized Adaptive CycleGAN [44.34422859532988]
  Unpaired image-to-image translation refers to learning inter-image-domain mapping in an unsupervised manner. Existing methods often learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty. We propose a novel probabilistic method called Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC)
  arXiv  Detail & Related papers  (2021-02-23T15:22:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.