A structured regression approach for evaluating model performance across intersectional subgroups

• URL: http://arxiv.org/abs/2401.14893v2
• Date: Tue, 14 May 2024 16:32:08 GMT
• Title: A structured regression approach for evaluating model performance across intersectional subgroups
• Authors: Christine Herlihy, Kimberly Truong, Alexandra Chouldechova, Miroslav Dudik,
• Abstract summary: Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
• Score: 53.91682617836498
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups defined by combinations of demographic or other sensitive attributes. The standard approach is to stratify the evaluation data across subgroups and compute performance metrics separately for each group. However, even for moderately-sized evaluation datasets, sample sizes quickly get small once considering intersectional subgroups, which greatly limits the extent to which intersectional groups are included in analysis. In this work, we introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups. We provide corresponding inference strategies for constructing confidence intervals and explore how goodness-of-fit testing can yield insight into the structure of fairness-related harms experienced by intersectional groups. We evaluate our approach on two publicly available datasets, and several variants of semi-synthetic data. The results show that our method is considerably more accurate than the standard approach, especially for small subgroups, and demonstrate how goodness-of-fit testing helps identify the key factors that drive differences in performance.

Related papers

• Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing [24.553384023323332]
  We propose an approach to test for performance disparities based on Conditional Value-at-Risk. We show that the sample complexity required for discovering performance violations is reduced exponentially to be at most upper bounded by the square root of the number of groups.
  arXiv  Detail & Related papers  (2023-12-06T19:25:32Z)
• Leveraging Structure for Improved Classification of Grouped Biased Data [8.121462458089143]
  We consider semi-supervised binary classification for applications in which data points are naturally grouped. We derive a semi-supervised algorithm that explicitly leverages the structure to learn an optimal, group-aware, probability-outputd classifier.
  arXiv  Detail & Related papers  (2022-12-07T15:18:21Z)
• CEREAL: Few-Sample Clustering Evaluation [4.569028973407756]
  We focus on the underexplored problem of estimating clustering quality with limited labels. We introduce CEREAL, a comprehensive framework for few-sample clustering evaluation. Our results show that CEREAL reduces the area under the absolute error curve by up to 57% compared to the best sampling baseline.
  arXiv  Detail & Related papers  (2022-09-30T19:52:41Z)
• Rethinking Clustering-Based Pseudo-Labeling for Unsupervised Meta-Learning [146.11600461034746]
  Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling. This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data. We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
  arXiv  Detail & Related papers  (2022-09-27T19:04:36Z)
• A One-shot Framework for Distributed Clustered Learning in Heterogeneous Environments [54.172993875654015]
  The paper proposes a family of communication efficient methods for distributed learning in heterogeneous environments. One-shot approach, based on local computations at the users and a clustering based aggregation step at the server is shown to provide strong learning guarantees. For strongly convex problems it is shown that, as long as the number of data points per user is above a threshold, the proposed approach achieves order-optimal mean-squared error rates in terms of the sample size.
  arXiv  Detail & Related papers  (2022-09-22T09:04:10Z)
• The Group Loss++: A deeper look into group loss for deep metric learning [65.19665861268574]
  Group Loss is a loss function based on a differentiable label-propagation method that enforces embedding similarity across all samples of a group. We show state-of-the-art results on clustering and image retrieval on four datasets, and present competitive results on two person re-identification datasets.
  arXiv  Detail & Related papers  (2022-04-04T14:09:58Z)
• HAWKS: Evolving Challenging Benchmark Sets for Cluster Analysis [2.5329716878122404]
  Comprehensive benchmarking of clustering algorithms is difficult. There is no consensus regarding the best practice for rigorous benchmarking. We demonstrate the important role evolutionary algorithms play to support flexible generation of such benchmarks.
  arXiv  Detail & Related papers  (2021-02-13T15:01:34Z)
• Fairness with Overlapping Groups [15.154984899546333]
  A standard goal is to ensure the equality of fairness metrics across multiple overlapping groups simultaneously. We reconsider this standard fair classification problem using a probabilistic population analysis. Our approach unifies a variety of existing group-fair classification methods and enables extensions to a wide range of non-decomposable multiclass performance metrics and fairness measures.
  arXiv  Detail & Related papers  (2020-06-24T05:01:10Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)
• Robust Recursive Partitioning for Heterogeneous Treatment Effects with Uncertainty Quantification [84.53697297858146]
  Subgroup analysis of treatment effects plays an important role in applications from medicine to public policy to recommender systems. Most of the current methods of subgroup analysis begin with a particular algorithm for estimating individualized treatment effects (ITE) This paper develops a new method for subgroup analysis, R2P, that addresses all these weaknesses.
  arXiv  Detail & Related papers  (2020-06-14T14:50:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.