Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs

• URL: http://arxiv.org/abs/2602.03895v1
• Date: Tue, 03 Feb 2026 08:37:37 GMT
• Title: Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs
• Authors: Xuwei Tan, Ziyu Hu, Xueru Zhang,
• Abstract summary: Machine learning models trained on real-world data often inherit and amplify biases against certain social groups.<n>We introduce NH-Fair, a unified benchmark for fairness without harm under standardized data, metrics, and training protocols.
• Score: 14.88523903012028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models trained on real-world data often inherit and amplify biases against certain social groups, raising urgent concerns about their deployment at scale. While numerous bias mitigation methods have been proposed, comparing the effectiveness of bias mitigation methods remains difficult due to heterogeneous datasets, inconsistent fairness metrics, isolated evaluation of vision versus multi-modal models, and insufficient hyperparameter tuning that undermines fair comparisons. We introduce NH-Fair, a unified benchmark for fairness without harm that spans both vision models and large vision-language models (LVLMs) under standardized data, metrics, and training protocols, covering supervised and zero-shot regimes. Our key contributions are: (1) a systematic ERM tuning study that identifies training choices with large influence on both utility and disparities, yielding empirically grounded guidelines to help practitioners reduce expensive hyperparameter tuning space in achieving strong fairness and accuracy; (2) evidence that many debiasing methods do not reliably outperform a well-tuned ERM baseline, whereas a composite data-augmentation method consistently delivers parity gains without sacrificing utility, emerging as a promising practical strategy. (3) an analysis showing that while LVLMs achieve higher average accuracy, they still exhibit subgroup disparities, and gains from scaling are typically smaller than those from architectural or training-protocol choices. NH-Fair provides a reproducible, tuning-aware pipeline for rigorous, harm-aware fairness evaluation.

Related papers

• Fair Bayesian Data Selection via Generalized Discrepancy Measures [11.013077130984973]
  We propose a data selection framework that ensures fairness by aligning group-specific posterior distributions of model parameters and sample weights with a shared central distribution.<n>Our framework supports flexible alignment via various distributional discrepancy measures, including Wasserstein distance, maximum mean discrepancy, and $f$-divergence.<n> Experiments on benchmark datasets show that our method consistently outperforms existing data selection and model-based fairness methods in both fairness and accuracy.
  arXiv  Detail & Related papers  (2025-11-10T12:28:04Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Fairness-Aware Low-Rank Adaptation Under Demographic Privacy Constraints [4.647881572951815]
  Pre-trained foundation models can be adapted for specific tasks using Low-Rank Adaptation (LoRA)<n>Existing fairness-aware fine-tuning methods rely on direct access to sensitive attributes or their predictors.<n>We introduce a set of LoRA-based fine-tuning methods that can be trained in a distributed fashion.
  arXiv  Detail & Related papers  (2025-03-07T18:49:57Z)
• FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation [3.959853359438669]
  We introduce FairLoRA, a novel fairness-specific regularizer for Low Rank Adaptation (LoRA) Our results demonstrate that the need for higher ranks to mitigate bias is not universal; it depends on factors such as the pre-trained model, dataset, and task.
  arXiv  Detail & Related papers  (2024-10-22T18:50:36Z)
• Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibrium [0.3350491650545292]
  Current methods for mitigating bias often result in information loss and an inadequate balance between accuracy and fairness. We propose a novel methodology grounded in bilevel optimization principles. Our deep learning-based approach concurrently optimize for both accuracy and fairness objectives.
  arXiv  Detail & Related papers  (2024-10-21T18:53:39Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Marginal Debiased Network for Fair Visual Recognition [59.05212866862219]
  We propose a novel marginal debiased network (MDN) to learn debiased representations. Our MDN can achieve a remarkable performance on under-represented samples.
  arXiv  Detail & Related papers  (2024-01-04T08:57:09Z)
• Non-Invasive Fairness in Learning through the Lens of Data Drift [88.37640805363317]
  We show how to improve the fairness of Machine Learning models without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data.
  arXiv  Detail & Related papers  (2023-03-30T17:30:42Z)
• Fairness Reprogramming [42.65700878967251]
  We propose a new generic fairness learning paradigm, called FairReprogram, which incorporates the model reprogramming technique. Specifically, FairReprogram considers the case where models can not be changed and appends to the input a set of perturbations, called the fairness trigger. We show both theoretically and empirically that the fairness trigger can effectively obscure demographic biases in the output prediction of fixed ML models.
  arXiv  Detail & Related papers  (2022-09-21T09:37:00Z)
• Normalise for Fairness: A Simple Normalisation Technique for Fairness in Regression Machine Learning Problems [46.93320580613236]
  We present a simple, yet effective method based on normalisation (FaiReg) for regression problems. We compare it with two standard methods for fairness, namely data balancing and adversarial training. The results show the superior performance of diminishing the effects of unfairness better than data balancing.
  arXiv  Detail & Related papers  (2022-02-02T12:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.