FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

• URL: http://arxiv.org/abs/2603.26008v1
• Date: Fri, 27 Mar 2026 01:55:14 GMT
• Title: FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants
• Abstract summary: Multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks.<n>We introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning.<n>Experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities.
• Score: 12.849143597169801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While powerful in image-conditioned generation, multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks. In safety-critical clinical settings, such disparities risk producing unequal diagnostic narratives and eroding trust in AI-assisted decision-making. While fairness has been studied extensively in vision-only and language-only models, its impact on MLLMs remains largely underexplored. To address these biases, we introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning without compromising overall performance. By minimizing the mutual information between target attributes, FairLLaVA regularizes the model's representations to be demographic-invariant. The method can be incorporated as a lightweight plug-in, maintaining efficiency with low-rank adapter fine-tuning, and provides an architecture-agnostic approach to fair visual instruction following. Extensive experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities while improving both equity-scaled clinical performance and natural language generation quality across diverse medical imaging modalities. Code can be accessed at https://github.com/bhosalems/FairLLaVA.

Related papers

• From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model [81.67708143891319]
  Vision-language models (VLMs) have demonstrated remarkable zero-shot capabilities yet remain sensitive to real-world distribution shifts during inference.<n>We bridge the detached objectives of inference and adaptation forVLMs, and propose a principled VLM TTA method called algname.<n>Experiments demonstrate the effectiveness and efficiency of our methods, outperforming the best-performing methods by up to 7% with state-of-the-art efficiency.
  arXiv  Detail & Related papers  (2026-08-18T21:44:32Z)
• Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement [50.97806700110183]
  Failure-informed Image Self-Augmentation (textbfFISA) is a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases.<n>Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.
  arXiv  Detail & Related papers  (2026-08-04T14:28:42Z)
• Fairness Beyond Demographics: Optimizing Performance Across Appearance-Based Hidden Cohorts in Medical Imaging [9.646042362182866]
  We introduce the label-free hidden-cohort fairness (LHCF) training paradigm for medical image analysis.<n>LHCF focuses on fairness across latent subpopulations discovered from image appearance.<n>We demonstrate on our proposed fairness benchmark, HIDFairBench, that LHCF provides state-of-the-art fairness results on single and multiple demographic attributes.
  arXiv  Detail & Related papers  (2026-05-28T12:07:24Z)
• FairEnc: A Fair Vision-Language Model with Fair Vision and Text Encoders for Glaucoma Detection [3.244353792776956]
  We propose FairEnc, a fair pretraining method for vision-language models (VLMs)<n>FairEnc mitigates biases in both textual and visual modalities with respect to sensitive attributes, including race, gender, ethnicity, and language.<n>Experiments on the publicly available Harvard-FairVLMed dataset demonstrate that FairEnc effectively reduces demographic disparity.
  arXiv  Detail & Related papers  (2026-05-06T13:18:34Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation [51.743225614196774]
  Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning.<n>They remain vulnerable to hallucination, where generated content deviates from visual evidence.<n>Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding.<n>We propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs.
  arXiv  Detail & Related papers  (2026-02-27T14:18:51Z)
• Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs [14.88523903012028]
  Machine learning models trained on real-world data often inherit and amplify biases against certain social groups.<n>We introduce NH-Fair, a unified benchmark for fairness without harm under standardized data, metrics, and training protocols.
  arXiv  Detail & Related papers  (2026-02-03T08:37:37Z)
• Fairness in Multi-modal Medical Diagnosis with Demonstration Selection [45.767489124851814]
  We propose Fairness-Aware Demonstration Selection (FADS), which builds demographically balanced and semantically relevant demonstrations.<n>FADS consistently reduces gender-, race-, and ethnicity-related disparities while maintaining strong accuracy.<n>These results highlight the potential of fairness-aware in-context learning as a scalable and data-efficient solution for equitable medical image reasoning.
  arXiv  Detail & Related papers  (2025-11-20T02:38:00Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.