A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes

• URL: http://arxiv.org/abs/2410.13286v2
• Date: Mon, 21 Oct 2024 08:00:06 GMT
• Title: A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes
• Authors: Jake Robertson, Thorsten Schmidt, Frank Hutter, Noor Awad,
• Abstract summary: ManyFairHPO is a fairness-aware model selection framework that enables practitioners to navigate complex and nuanced fairness objective landscapes. We demonstrate the effectiveness of ManyFairHPO in balancing multiple fairness objectives, mitigating risks such as self-fulfilling prophecies, and providing interpretable insights to guide stakeholders in making fairness-aware modeling decisions.
• Score: 37.5215569371757
• License:
• Abstract: Fairness-aware Machine Learning (FairML) applications are often characterized by complex social objectives and legal requirements, frequently involving multiple, potentially conflicting notions of fairness. Despite the well-known Impossibility Theorem of Fairness and extensive theoretical research on the statistical and socio-technical trade-offs between fairness metrics, many FairML tools still optimize or constrain for a single fairness objective. However, this one-sided optimization can inadvertently lead to violations of other relevant notions of fairness. In this socio-technical and empirical study, we frame fairness as a many-objective (MaO) problem by treating fairness metrics as conflicting objectives. We introduce ManyFairHPO, a human-in-the-loop, fairness-aware model selection framework that enables practitioners to effectively navigate complex and nuanced fairness objective landscapes. ManyFairHPO aids in the identification, evaluation, and balancing of fairness metric conflicts and their related social consequences, leading to more informed and socially responsible model-selection decisions. Through a comprehensive empirical evaluation and a case study on the Law School Admissions problem, we demonstrate the effectiveness of ManyFairHPO in balancing multiple fairness objectives, mitigating risks such as self-fulfilling prophecies, and providing interpretable insights to guide stakeholders in making fairness-aware modeling decisions.

Related papers

• The Impossibility of Fair LLMs [59.424918263776284]
  The need for fair AI is increasingly clear in the era of large language models (LLMs) We review the technical frameworks that machine learning researchers have used to evaluate fairness. We develop guidelines for the more realistic goal of achieving fairness in particular use cases.
  arXiv  Detail & Related papers  (2024-05-28T04:36:15Z)
• Fair Few-shot Learning with Auxiliary Sets [53.30014767684218]
  In many machine learning (ML) tasks, only very few labeled data samples can be collected, which can lead to inferior fairness performance. In this paper, we define the fairness-aware learning task with limited training samples as the emphfair few-shot learning problem. We devise a novel framework that accumulates fairness-aware knowledge across different meta-training tasks and then generalizes the learned knowledge to meta-test tasks.
  arXiv  Detail & Related papers  (2023-08-28T06:31:37Z)
• Learning for Counterfactual Fairness from Observational Data [62.43249746968616]
  Fairness-aware machine learning aims to eliminate biases of learning models against certain subgroups described by certain protected (sensitive) attributes such as race, gender, and age. A prerequisite for existing methods to achieve counterfactual fairness is the prior human knowledge of the causal model for the data. In this work, we address the problem of counterfactually fair prediction from observational data without given causal models by proposing a novel framework CLAIRE.
  arXiv  Detail & Related papers  (2023-07-17T04:08:29Z)
• DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision [73.80009454050858]
  This work presents a self-supervised model, called DualFair, that can debias sensitive attributes like gender and race from learned representations. Our model jointly optimize for two fairness criteria - group fairness and counterfactual fairness.
  arXiv  Detail & Related papers  (2023-03-15T07:13:54Z)
• Causal Fairness Analysis [68.12191782657437]
  We introduce a framework for understanding, modeling, and possibly solving issues of fairness in decision-making settings. The main insight of our approach will be to link the quantification of the disparities present on the observed data with the underlying, and often unobserved, collection of causal mechanisms. Our effort culminates in the Fairness Map, which is the first systematic attempt to organize and explain the relationship between different criteria found in the literature.
  arXiv  Detail & Related papers  (2022-07-23T01:06:34Z)
• Algorithmic Fairness in Business Analytics: Directions for Research and Practice [24.309795052068388]
  This paper offers a forward-looking, BA-focused review of algorithmic fairness. We first review the state-of-the-art research on sources and measures of bias, as well as bias mitigation algorithms. We then provide a detailed discussion of the utility-fairness relationship, emphasizing that the frequent assumption of a trade-off between these two constructs is often mistaken or short-sighted.
  arXiv  Detail & Related papers  (2022-07-22T10:21:38Z)
• What Is Fairness? On the Role of Protected Attributes and Fictitious Worlds [8.223468651994352]
  A growing body of literature in fairness-aware machine learning (fairML) aims to mitigate machine learning (ML)-related unfairness in automated decision-making (ADM) However, the underlying concept of fairness is rarely discussed, leaving a significant gap between centuries of philosophical discussion and the recent adoption of the concept in the ML community. We try to bridge this gap by formalizing a consistent concept of fairness and by translating the philosophical considerations into a formal framework for the training and evaluation of ML models in ADM systems.
  arXiv  Detail & Related papers  (2022-05-19T15:37:26Z)
• Fairness in Machine Learning [15.934879442202785]
  We show how causal Bayesian networks can play an important role to reason about and deal with fairness. We present a unified framework that encompasses methods that can deal with different settings and fairness criteria.
  arXiv  Detail & Related papers  (2020-12-31T18:38:58Z)
• Ethical Adversaries: Towards Mitigating Unfairness with Adversarial Machine Learning [8.436127109155008]
  Individuals, as well as organisations, notice, test, and criticize unfair results to hold model designers and deployers accountable. We offer a framework that assists these groups in mitigating unfair representations stemming from the training datasets. Our framework relies on two inter-operating adversaries to improve fairness.
  arXiv  Detail & Related papers  (2020-05-14T10:10:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.