FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods

• URL: http://arxiv.org/abs/2306.09468v2
• Date: Tue, 11 Jun 2024 03:10:10 GMT
• Title: FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
• Authors: Xiaotian Han, Jianfeng Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, Xia Hu,
• Abstract summary: This paper introduces the Fair Fairness Benchmark (textsfFFB), a benchmarking framework for in-processing group fairness methods. We provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness.
• Score: 84.1077756698332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces the Fair Fairness Benchmark (\textsf{FFB}), a benchmarking framework for in-processing group fairness methods. Ensuring fairness in machine learning is important for ethical compliance. However, there exist challenges in comparing and developing fairness methods due to inconsistencies in experimental settings, lack of accessible algorithmic implementations, and limited extensibility of current fairness packages and tools. To address these issues, we introduce an open-source standardized benchmark for evaluating in-processing group fairness methods and provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness. This work offers the following key contributions: the provision of flexible, extensible, minimalistic, and research-oriented open-source code; the establishment of unified fairness method benchmarking pipelines; and extensive benchmarking, which yields key insights from $\mathbf{45,079}$ experiments, $\mathbf{14,428}$ GPU hours. We believe that our work will significantly facilitate the growth and development of the fairness research community.

Related papers

• Finite-Sample and Distribution-Free Fair Classification: Optimal Trade-off Between Excess Risk and Fairness, and the Cost of Group-Blindness [14.421493372559762]
  We quantify the impact of enforcing algorithmic fairness and group-blindness in binary classification under group fairness constraints. We propose a unified framework for fair classification that provides distribution-free and finite-sample fairness guarantees with controlled excess risk.
  arXiv  Detail & Related papers  (2024-10-21T20:04:17Z)
• A Benchmark for Fairness-Aware Graph Learning [58.515305543487386]
  We present an extensive benchmark on ten representative fairness-aware graph learning methods. Our in-depth analysis reveals key insights into the strengths and limitations of existing methods.
  arXiv  Detail & Related papers  (2024-07-16T18:43:43Z)
• Distribution-Free Fair Federated Learning with Small Samples [54.63321245634712]
  FedFaiREE is a post-processing algorithm developed specifically for distribution-free fair learning in decentralized settings with small samples. We provide rigorous theoretical guarantees for both fairness and accuracy, and our experimental results further provide robust empirical validation for our proposed method.
  arXiv  Detail & Related papers  (2024-02-25T17:37:53Z)
• FaiREE: Fair Classification with Finite-Sample and Distribution-Free Guarantee [40.10641140860374]
  FaiREE is a fair classification algorithm that can satisfy group fairness constraints with finite-sample and distribution-free theoretical guarantees. FaiREE is shown to have favorable performance over state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-11-28T05:16:20Z)
• How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts [107.72786199113183]
  We propose a novel fairness learning method termed CUrvature MAtching (CUMA) CUMA achieves robust fairness generalizable to unseen domains with unknown distributional shifts. We evaluate our method on three popular fairness datasets.
  arXiv  Detail & Related papers  (2022-07-04T02:37:50Z)
• fairlib: A Unified Framework for Assessing and Improving Classification Fairness [66.27822109651757]
  fairlib is an open-source framework for assessing and improving classification fairness. We implement 14 debiasing methods, including pre-processing, at-training-time, and post-processing approaches. The built-in metrics cover the most commonly used fairness criterion and can be further generalized and customized for fairness evaluation.
  arXiv  Detail & Related papers  (2022-05-04T03:50:23Z)
• Fair Federated Learning via Bounded Group Loss [37.72259706322158]
  We propose a general framework for provably fair federated learning. We extend the notion of Bounded Group Loss as a theoretically-grounded approach for group fairness. We provide convergence guarantees for the method as well as fairness guarantees for the resulting solution.
  arXiv  Detail & Related papers  (2022-03-18T23:11:54Z)
• A Systematic Approach to Group Fairness in Automated Decision Making [0.0]
  The goal of this paper is to provide data scientists with an accessible introduction to group fairness metrics. We will do this by considering in which sense socio-demographic groups are compared for making a statement on fairness.
  arXiv  Detail & Related papers  (2021-09-09T12:47:15Z)
• Learning to Generate Fair Clusters from Demonstrations [27.423983748614198]
  We show how to identify the intended fairness constraint for a problem based on limited demonstrations from an expert. We present an algorithm to identify the fairness metric from demonstrations and generate clusters using existing off-the-shelf clustering techniques. We investigate how to generate interpretable solutions using our approach.
  arXiv  Detail & Related papers  (2021-02-08T03:09:33Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.