Retiring $\Delta$DP: New Distribution-Level Metrics for Demographic Parity

• URL: http://arxiv.org/abs/2301.13443v3
• Date: Sat, 10 Jun 2023 03:13:29 GMT
• Title: Retiring $\Delta$DP: New Distribution-Level Metrics for Demographic Parity
• Authors: Xiaotian Han, Zhimeng Jiang, Hongye Jin, Zirui Liu, Na Zou, Qifan Wang, Xia Hu
• Abstract summary: The fairness metric $Delta DP$ can not precisely measure the violation of demographic parity. We propose two new fairness metrics, Area Between Probability density function Curves (ABPC) and Area Between Cumulative density function Curves (ABCC) Our proposed new metrics enjoy: i) zero-value ABCC/ABPC guarantees zero violation of demographic parity; ii) ABCC/ABPC guarantees demographic parity while the classification thresholds are adjusted.
• Score: 47.78843764957511
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Demographic parity is the most widely recognized measure of group fairness in machine learning, which ensures equal treatment of different demographic groups. Numerous works aim to achieve demographic parity by pursuing the commonly used metric $\Delta DP$. Unfortunately, in this paper, we reveal that the fairness metric $\Delta DP$ can not precisely measure the violation of demographic parity, because it inherently has the following drawbacks: i) zero-value $\Delta DP$ does not guarantee zero violation of demographic parity, ii) $\Delta DP$ values can vary with different classification thresholds. To this end, we propose two new fairness metrics, Area Between Probability density function Curves (ABPC) and Area Between Cumulative density function Curves (ABCC), to precisely measure the violation of demographic parity at the distribution level. The new fairness metrics directly measure the difference between the distributions of the prediction probability for different demographic groups. Thus our proposed new metrics enjoy: i) zero-value ABCC/ABPC guarantees zero violation of demographic parity; ii) ABCC/ABPC guarantees demographic parity while the classification thresholds are adjusted. We further re-evaluate the existing fair models with our proposed fairness metrics and observe different fairness behaviors of those models under the new metrics. The code is available at https://github.com/ahxt/new_metric_for_demographic_parity

Related papers

• Score Normalization for Demographic Fairness in Face Recognition [16.421833444307232]
  Well-known sample-centered score normalization techniques, Z-norm and T-norm, do not improve fairness for high-security operating points. We extend the standard Z/T-norm to integrate demographic information in normalization. We show that our techniques generally improve the overall fairness of five state-of-the-art pre-trained face recognition networks.
  arXiv  Detail & Related papers  (2024-07-19T07:51:51Z)
• Individual Fairness Through Reweighting and Tuning [0.23395944472515745]
  Inherent bias within society can be amplified and perpetuated by artificial intelligence (AI) systems. Recently, Graph Laplacian Regularizer (GLR) has been used as a substitute for the common Lipschitz condition to enhance individual fairness. In this work, we investigated whether defining a GLR independently on the train and target data could maintain similar accuracy.
  arXiv  Detail & Related papers  (2024-05-02T20:15:25Z)
• Minimax Optimal Fair Classification with Bounded Demographic Disparity [28.936244976415484]
  This paper explores the statistical foundations of fair binary classification with two protected groups. We show that using a finite sample incurs additional costs due to the need to estimate group-specific acceptance thresholds. We propose FairBayes-DDP+, a group-wise thresholding method with an offset that we show attains the minimax lower bound.
  arXiv  Detail & Related papers  (2024-03-27T02:59:04Z)
• Equal Opportunity of Coverage in Fair Regression [50.76908018786335]
  We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. We propose Equal Opportunity of Coverage (EOC) that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level.
  arXiv  Detail & Related papers  (2023-11-03T21:19:59Z)
• Semidefinite programming on population clustering: a global analysis [0.6472434306724609]
  We consider the problem of partitioning a small data sample of size $n$ drawn from a mixture of $2$ sub-gaussian distributions. We are interested in the case that individual features are of low average quality $gamma$, and we want to use as few of them as possible to correctly partition the sample.
  arXiv  Detail & Related papers  (2023-01-01T04:52:25Z)
• Estimating Structural Disparities for Face Models [54.062512989859265]
  In machine learning, disparity metrics are often defined by measuring the difference in the performance or outcome of a model, across different sub-populations. We explore performing such analysis on computer vision models trained on human faces, and on tasks such as face attribute prediction and affect estimation.
  arXiv  Detail & Related papers  (2022-04-13T05:30:53Z)
• Balancing Biases and Preserving Privacy on Balanced Faces in the Wild [50.915684171879036]
  There are demographic biases present in current facial recognition (FR) models. We introduce our Balanced Faces in the Wild dataset to measure these biases across different ethnic and gender subgroups. We find that relying on a single score threshold to differentiate between genuine and imposters sample pairs leads to suboptimal results. We propose a novel domain adaptation learning scheme that uses facial features extracted from state-of-the-art neural networks.
  arXiv  Detail & Related papers  (2021-03-16T15:05:49Z)
• A First Step Towards Distribution Invariant Regression Metrics [1.370633147306388]
  In classification, it has been stated repeatedly that performance metrics like the F-Measure and Accuracy are highly dependent on the class distribution. We show that the same problem exists in regression. The distribution of odometry parameters in robotic applications can for example largely vary between different recording sessions. Here, we need regression algorithms that either perform equally well for all function values, or that focus on certain boundary regions like high speed.
  arXiv  Detail & Related papers  (2020-09-10T23:40:46Z)
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models [67.66144604972052]
  Federated learning (FL) is a promising way to use the computing power of mobile devices while maintaining privacy of users. We show that a critical issue that affects the test accuracy is the large gradient diversity of the models from different users. We propose a novel grouping-based model averaging method to replace the FedAvg averaging method.
  arXiv  Detail & Related papers  (2020-08-26T03:36:07Z)
• On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation [86.11292297348622]
  We show that a linear combination of quality and diversity constitutes a divergence metric between the generated distribution and the real distribution. We propose CR/NRR as a substitute for quality/diversity metric pair.
  arXiv  Detail & Related papers  (2020-07-03T04:06:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.