Testing Fairness with Utility Tradeoffs: A Wasserstein Projection Approach

• URL: http://arxiv.org/abs/2505.11678v3
• Date: Wed, 24 Sep 2025 12:20:53 GMT
• Title: Testing Fairness with Utility Tradeoffs: A Wasserstein Projection Approach
• Authors: Yan Chen, Zheng Tan, Jose Blanchet, Hanzhang Qin,
• Abstract summary: We propose a statistical hypothesis testing framework that jointly evaluates approximate fairness and utility.<n>Our framework builds on the strong demographic parity criterion and incorporates a utility measure motivated by the potential outcomes framework.<n>We show that the test is computationally tractable, interpretable, broadly applicable across machine learning models, and extendable to more general settings.
• Score: 6.378410364292642
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring fairness in data driven decision making has become a central concern across domains such as marketing, lending, and healthcare, but fairness constraints often come at the cost of utility. We propose a statistical hypothesis testing framework that jointly evaluates approximate fairness and utility, relaxing strict fairness requirements while ensuring that overall utility remains above a specified threshold. Our framework builds on the strong demographic parity (SDP) criterion and incorporates a utility measure motivated by the potential outcomes framework. The test statistic is constructed via Wasserstein projections, enabling auditors to assess whether observed fairness-utility tradeoffs are intrinsic to the algorithm or attributable to randomness in the data. We show that the test is computationally tractable, interpretable, broadly applicable across machine learning models, and extendable to more general settings. We apply our approach to multiple real-world datasets, offering new insights into the fairness-utility tradeoff through the perspective of statistical hypothesis testing.

Related papers

• Fair Bayesian Data Selection via Generalized Discrepancy Measures [11.013077130984973]
  We propose a data selection framework that ensures fairness by aligning group-specific posterior distributions of model parameters and sample weights with a shared central distribution.<n>Our framework supports flexible alignment via various distributional discrepancy measures, including Wasserstein distance, maximum mean discrepancy, and $f$-divergence.<n> Experiments on benchmark datasets show that our method consistently outperforms existing data selection and model-based fairness methods in both fairness and accuracy.
  arXiv  Detail & Related papers  (2025-11-10T12:28:04Z)
• Reliable and Reproducible Demographic Inference for Fairness in Face Analysis [63.46525489354455]
  We propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach.<n>We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency.<n>Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute.
  arXiv  Detail & Related papers  (2025-10-23T12:22:02Z)
• Beyond Internal Data: Bounding and Estimating Fairness from Incomplete Data [26.037607208689977]
  In high-stakes domains such as lending, hiring, and healthcare, ensuring fairness in AI systems is critical.<n>In industry settings, legal and privacy concerns restrict the collection of demographic data required to assess group disparities.<n>Our work seeks to leverage such available separate data to estimate model fairness when complete data is inaccessible.
  arXiv  Detail & Related papers  (2025-08-18T15:57:30Z)
• Beyond Internal Data: Constructing Complete Datasets for Fairness Testing [26.037607208689977]
  This work focuses on evaluating classifier fairness when complete datasets including demographics are inaccessible.<n>We propose leveraging separate overlapping datasets to construct complete synthetic data that includes demographic information.<n>We validate the fidelity of the synthetic data by comparing it to real data, and empirically demonstrate that fairness metrics derived from testing on such synthetic data are consistent with those obtained from real data.
  arXiv  Detail & Related papers  (2025-07-24T16:35:42Z)
• Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness [49.35494016290887]
  We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of relevant populations but reflective of real-world disparities.<n>Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift.
  arXiv  Detail & Related papers  (2025-06-04T17:40:31Z)
• Algorithmic Tradeoffs in Fair Lending: Profitability, Compliance, and Long-Term Impact [0.0]
  We quantify how different fairness interventions impact profit margins and default rates.<n>We identify the specific economic conditions under which fair lending becomes profitable.<n>These findings offer practical guidance for designing lending algorithms that balance ethical considerations with business objectives.
  arXiv  Detail & Related papers  (2025-05-08T19:18:33Z)
• Fairness-aware organ exchange and kidney paired donation [10.277630436997365]
  The kidney paired donation (KPD) program provides an innovative solution to overcome incompatibility challenges in kidney transplants.<n>To address unequal access to transplant opportunities, there are two widely used fairness criteria: group fairness and individual fairness.<n>Motivated by the calibration principle in machine learning, we introduce a new fairness criterion: the matching outcome should be conditionally independent of the protected feature.
  arXiv  Detail & Related papers  (2025-03-09T04:01:08Z)
• Targeted Learning for Data Fairness [52.59573714151884]
  We expand fairness inference by evaluating fairness in the data generating process itself.<n>We derive estimators demographic parity, equal opportunity, and conditional mutual information.<n>To validate our approach, we perform several simulations and apply our estimators to real data.
  arXiv  Detail & Related papers  (2025-02-06T18:51:28Z)
• Thinking Racial Bias in Fair Forgery Detection: Models, Datasets and Evaluations [63.52709761339949]
  We first contribute a dedicated dataset called the Fair Forgery Detection (FairFD) dataset, where we prove the racial bias of public state-of-the-art (SOTA) methods.<n>We design novel metrics including Approach Averaged Metric and Utility Regularized Metric, which can avoid deceptive results.<n>We also present an effective and robust post-processing technique, Bias Pruning with Fair Activations (BPFA), which improves fairness without requiring retraining or weight updates.
  arXiv  Detail & Related papers  (2024-07-19T14:53:18Z)
• Fairness-Accuracy Trade-Offs: A Causal Perspective [58.06306331390586]
  We analyze the tension between fairness and accuracy from a causal lens for the first time.<n>We show that enforcing a causal constraint often reduces the disparity between demographic groups.<n>We introduce a new neural approach for causally-constrained fair learning.
  arXiv  Detail & Related papers  (2024-05-24T11:19:52Z)
• The Unfairness of $\varepsilon$-Fairness [0.0]
  We show that if the concept of $varepsilon$-fairness is employed, it can possibly lead to outcomes that are maximally unfair in the real-world context. We illustrate our findings with two real-world examples: college admissions and credit risk assessment.
  arXiv  Detail & Related papers  (2024-05-15T14:13:35Z)
• Achievable Fairness on Your Data With Utility Guarantees [16.78730663293352]
  In machine learning fairness, training models that minimize disparity across different sensitive groups often leads to diminished accuracy. We present a computationally efficient approach to approximate the fairness-accuracy trade-off curve tailored to individual datasets. We introduce a novel methodology for quantifying uncertainty in our estimates, thereby providing practitioners with a robust framework for auditing model fairness.
  arXiv  Detail & Related papers  (2024-02-27T00:59:32Z)
• FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods [84.1077756698332]
  This paper introduces the Fair Fairness Benchmark (textsfFFB), a benchmarking framework for in-processing group fairness methods. We provide a comprehensive analysis of state-of-the-art methods to ensure different notions of group fairness.
  arXiv  Detail & Related papers  (2023-06-15T19:51:28Z)
• Causal Fairness for Outcome Control [68.12191782657437]
  We study a specific decision-making task called outcome control in which an automated system aims to optimize an outcome variable $Y$ while being fair and equitable. In this paper, we first analyze through causal lenses the notion of benefit, which captures how much a specific individual would benefit from a positive decision. We then note that the benefit itself may be influenced by the protected attribute, and propose causal tools which can be used to analyze this.
  arXiv  Detail & Related papers  (2023-06-08T09:31:18Z)
• Reconciling Predictive and Statistical Parity: A Causal Approach [68.59381759875734]
  We propose a new causal decomposition formula for the fairness measures associated with predictive parity. We show that the notions of statistical and predictive parity are not really mutually exclusive, but complementary and spanning a spectrum of fairness notions.
  arXiv  Detail & Related papers  (2023-06-08T09:23:22Z)
• Fair and Robust Estimation of Heterogeneous Treatment Effects for Policy Learning [2.356908851188234]
  We use this framework to characterize the trade-off between fairness and the maximum welfare achievable by the optimal policy. We evaluate the methods in a simulation study and illustrate them in a real-world case study.
  arXiv  Detail & Related papers  (2023-06-06T12:22:20Z)
• Fairness in Matching under Uncertainty [78.39459690570531]
  algorithmic two-sided marketplaces have drawn attention to the issue of fairness in such settings. We axiomatize a notion of individual fairness in the two-sided marketplace setting which respects the uncertainty in the merits. We design a linear programming framework to find fair utility-maximizing distributions over allocations.
  arXiv  Detail & Related papers  (2023-02-08T00:30:32Z)
• Group Fairness in Prediction-Based Decision Making: From Moral Assessment to Implementation [0.0]
  We introduce a framework for the moral assessment of what fairness means in a given context. We map the assessment's results to established statistical group fairness criteria. We extend the FEC principle to cover all types of group fairness criteria.
  arXiv  Detail & Related papers  (2022-10-19T10:44:21Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Beyond Individual and Group Fairness [90.4666341812857]
  We present a new data-driven model of fairness that is guided by the unfairness complaints received by the system. Our model supports multiple fairness criteria and takes into account their potential incompatibilities.
  arXiv  Detail & Related papers  (2020-08-21T14:14:44Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.