Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML

• URL: http://arxiv.org/abs/2411.11101v2
• Date: Tue, 19 Nov 2024 02:39:53 GMT
• Title: Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML
• Authors: Prakhar Ganesh, Usman Gohar, Lu Cheng, Golnoosh Farnadi,
• Abstract summary: We show significant variance in fairness achieved by several algorithms and the influence of the learning pipeline on fairness scores. We highlight that most bias mitigation techniques can achieve comparable performance. We hope our work encourages future research on how various choices in the lifecycle of developing an algorithm impact fairness.
• Score: 9.579645248339004
• License:
• Abstract: With fairness concerns gaining significant attention in Machine Learning (ML), several bias mitigation techniques have been proposed, often compared against each other to find the best method. These benchmarking efforts tend to use a common setup for evaluation under the assumption that providing a uniform environment ensures a fair comparison. However, bias mitigation techniques are sensitive to hyperparameter choices, random seeds, feature selection, etc., meaning that comparison on just one setting can unfairly favour certain algorithms. In this work, we show significant variance in fairness achieved by several algorithms and the influence of the learning pipeline on fairness scores. We highlight that most bias mitigation techniques can achieve comparable performance, given the freedom to perform hyperparameter optimization, suggesting that the choice of the evaluation parameters-rather than the mitigation technique itself-can sometimes create the perceived superiority of one method over another. We hope our work encourages future research on how various choices in the lifecycle of developing an algorithm impact fairness, and trends that guide the selection of appropriate algorithms.

Related papers

• ABCFair: an Adaptable Benchmark approach for Comparing Fairness Methods [12.774108753281809]
  We introduce ABCFair, a benchmark approach which allows adapting to the desiderata of the real-world problem setting. We apply ABCFair to a range of pre-, in-, and postprocessing methods on both large-scale, traditional datasets and on a dual label (biased and unbiased) dataset.
  arXiv  Detail & Related papers  (2024-09-25T14:26:07Z)
• Boosting Fair Classifier Generalization through Adaptive Priority Reweighing [59.801444556074394]
  A performance-promising fair algorithm with better generalizability is needed. This paper proposes a novel adaptive reweighing method to eliminate the impact of the distribution shifts between training and test data on model generalizability.
  arXiv  Detail & Related papers  (2023-09-15T13:04:55Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Are metaheuristics worth it? A computational comparison between nature-inspired and deterministic techniques on black-box optimization problems [0.0]
  In this paper, we provide an extensive computational comparison of selected methods from each of these branches. The results showed that, when dealing with situations where the objective function evaluations are relatively cheap, the nature-inspired methods have a significantly better performance than their deterministic counterparts.
  arXiv  Detail & Related papers  (2022-12-13T19:44:24Z)
• Adaptive Sampling for Heterogeneous Rank Aggregation from Noisy Pairwise Comparisons [85.5955376526419]
  In rank aggregation problems, users exhibit various accuracy levels when comparing pairs of items. We propose an elimination-based active sampling strategy, which estimates the ranking of items via noisy pairwise comparisons. We prove that our algorithm can return the true ranking of items with high probability.
  arXiv  Detail & Related papers  (2021-10-08T13:51:55Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• A Comparative Evaluation of Quantification Methods [3.1499058381005227]
  Quantification represents the problem of predicting class distributions in a dataset. A large variety of different algorithms has been proposed in recent years. We compare 24 different methods on overall more than 40 data sets.
  arXiv  Detail & Related papers  (2021-03-04T18:51:06Z)
• One-vs.-One Mitigation of Intersectional Bias: A General Method to Extend Fairness-Aware Binary Classification [0.48733623015338234]
  One-vs.-One Mitigation is a process of comparison between each pair of subgroups related to sensitive attributes to the fairness-aware machine learning for binary classification. Our method mitigates the intersectional bias much better than conventional methods in all the settings.
  arXiv  Detail & Related papers  (2020-10-26T11:35:39Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Learning the Truth From Only One Side of the Story [58.65439277460011]
  We focus on generalized linear models and show that without adjusting for this sampling bias, the model may converge suboptimally or even fail to converge to the optimal solution. We propose an adaptive approach that comes with theoretical guarantees and show that it outperforms several existing methods empirically.
  arXiv  Detail & Related papers  (2020-06-08T18:20:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.