Counterpart Fairness -- Addressing Systematic between-group Differences
in Fairness Evaluation
- URL: http://arxiv.org/abs/2305.18160v2
- Date: Mon, 28 Aug 2023 15:59:26 GMT
- Title: Counterpart Fairness -- Addressing Systematic between-group Differences
in Fairness Evaluation
- Authors: Yifei Wang, Zhengyang Zhou, Liqin Wang, John Laurentiev, Peter Hou, Li
Zhou, Pengyu Hong
- Abstract summary: We develop a propensity-score-based method for identifying counterparts, which prevents fairness evaluation from comparing "oranges" with "apples"
We propose a counterpart-based statistical fairness index, termed Counterpart-Fairness (CFair), to assess fairness of machine learning models.
- Score: 18.372355677006965
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: When using machine learning (ML) to aid decision-making, it is critical to
ensure that an algorithmic decision is fair, i.e., it does not discriminate
against specific individuals/groups, particularly those from underprivileged
populations. Existing group fairness methods require equal group-wise measures,
which however fails to consider systematic between-group differences. The
confounding factors, which are non-sensitive variables but manifest systematic
differences, can significantly affect fairness evaluation. To tackle this
problem, we believe that a fairness measurement should be based on the
comparison between counterparts (i.e., individuals who are similar to each
other with respect to the task of interest) from different groups, whose group
identities cannot be distinguished algorithmically by exploring confounding
factors. We have developed a propensity-score-based method for identifying
counterparts, which prevents fairness evaluation from comparing "oranges" with
"apples". In addition, we propose a counterpart-based statistical fairness
index, termed Counterpart-Fairness (CFair), to assess fairness of ML models.
Various empirical studies were conducted to validate the effectiveness of
CFair. We publish our code at \url{https://github.com/zhengyjo/CFair}.
Related papers
- A Canonical Data Transformation for Achieving Inter- and Within-group Fairness [17.820200610132265]
We introduce a formal definition of within-group fairness that maintains fairness among individuals from within the same group.
We propose a pre-processing framework to meet both inter- and within-group fairness criteria with little compromise in accuracy.
We apply this framework to the COMPAS risk assessment and Law School datasets and compare its performance to two regularization-based methods.
arXiv Detail & Related papers (2023-10-23T17:00:20Z) - Evaluating the Fairness of Discriminative Foundation Models in Computer
Vision [51.176061115977774]
We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP)
We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy.
Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
arXiv Detail & Related papers (2023-10-18T10:32:39Z) - On The Impact of Machine Learning Randomness on Group Fairness [11.747264308336012]
We investigate the impact on group fairness of different sources of randomness in training neural networks.
We show that the variance in group fairness measures is rooted in the high volatility of the learning process on under-represented groups.
We show how one can control group-level accuracy, with high efficiency and negligible impact on the model's overall performance, by simply changing the data order for a single epoch.
arXiv Detail & Related papers (2023-07-09T09:36:31Z) - Error Parity Fairness: Testing for Group Fairness in Regression Tasks [5.076419064097733]
This work presents error parity as a regression fairness notion and introduces a testing methodology to assess group fairness.
It is followed by a suitable permutation test to compare groups on several statistics to explore disparities and identify impacted groups.
Overall, the proposed regression fairness testing methodology fills a gap in the fair machine learning literature and may serve as a part of larger accountability assessments and algorithm audits.
arXiv Detail & Related papers (2022-08-16T17:47:20Z) - Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features.
We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
arXiv Detail & Related papers (2022-03-16T15:00:33Z) - Fair Group-Shared Representations with Normalizing Flows [68.29997072804537]
We develop a fair representation learning algorithm which is able to map individuals belonging to different groups in a single group.
We show experimentally that our methodology is competitive with other fair representation learning algorithms.
arXiv Detail & Related papers (2022-01-17T10:49:49Z) - Measuring Fairness Under Unawareness of Sensitive Attributes: A
Quantification-Based Approach [131.20444904674494]
We tackle the problem of measuring group fairness under unawareness of sensitive attributes.
We show that quantification approaches are particularly suited to tackle the fairness-under-unawareness problem.
arXiv Detail & Related papers (2021-09-17T13:45:46Z) - MultiFair: Multi-Group Fairness in Machine Learning [52.24956510371455]
We study multi-group fairness in machine learning (MultiFair)
We propose a generic end-to-end algorithmic framework to solve it.
Our proposed framework is generalizable to many different settings.
arXiv Detail & Related papers (2021-05-24T02:30:22Z) - Metric-Free Individual Fairness with Cooperative Contextual Bandits [17.985752744098267]
Group fairness requires that different groups should be treated similarly which might be unfair to some individuals within a group.
Individual fairness remains understudied due to its reliance on problem-specific similarity metrics.
We propose a metric-free individual fairness and a cooperative contextual bandits algorithm.
arXiv Detail & Related papers (2020-11-13T03:10:35Z) - Distributional Individual Fairness in Clustering [7.303841123034983]
We introduce a framework for assigning individuals, embedded in a metric space, to probability distributions over a bounded number of cluster centers.
We provide an algorithm for clustering with $p$-norm objective and individual fairness constraints with provable approximation guarantee.
arXiv Detail & Related papers (2020-06-22T20:02:09Z) - Algorithmic Decision Making with Conditional Fairness [48.76267073341723]
We define conditional fairness as a more sound fairness metric by conditioning on the fairness variables.
We propose a Derivable Conditional Fairness Regularizer (DCFR) to track the trade-off between precision and fairness of algorithmic decision making.
arXiv Detail & Related papers (2020-06-18T12:56:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.