Disclosure and Evaluation as Fairness Interventions for General-Purpose AI

• URL: http://arxiv.org/abs/2510.05292v1
• Date: Mon, 06 Oct 2025 19:00:42 GMT
• Title: Disclosure and Evaluation as Fairness Interventions for General-Purpose AI
• Authors: Vyoma Raman, Judy Hanwen Shen, Andy K. Zhang, Lindsey Gailmard, Rishi Bommasani, Daniel E. Ho, Angelina Wang,
• Abstract summary: We argue that while we cannot be prescriptive about what constitutes fair outcomes, we can specify the processes that different stakeholders should follow in service of fairness.<n>We consider the obligations of two major groups: system providers and system deployers.
• Score: 16.220252808413086
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite conflicting definitions and conceptions of fairness, AI fairness researchers broadly agree that fairness is context-specific. However, when faced with general-purpose AI, which by definition serves a range of contexts, how should we think about fairness? We argue that while we cannot be prescriptive about what constitutes fair outcomes, we can specify the processes that different stakeholders should follow in service of fairness. Specifically, we consider the obligations of two major groups: system providers and system deployers. While system providers are natural candidates for regulatory attention, the current state of AI understanding offers limited insight into how upstream factors translate into downstream fairness impacts. Thus, we recommend that providers invest in evaluative research studying how model development decisions influence fairness and disclose whom they are serving their models to, or at the very least, reveal sufficient information for external researchers to conduct such research. On the other hand, system deployers are closer to real-world contexts and can leverage their proximity to end users to address fairness harms in different ways. Here, we argue they should responsibly disclose information about users and personalization and conduct rigorous evaluations across different levels of fairness. Overall, instead of focusing on enforcing fairness outcomes, we prioritize intentional information-gathering by system providers and deployers that can facilitate later context-aware action. This allows us to be specific and concrete about the processes even while the contexts remain unknown. Ultimately, this approach can sharpen how we distribute fairness responsibilities and inform more fluid, context-sensitive interventions as AI continues to advance.

Related papers

• A Unifying Human-Centered AI Fairness Framework [2.9385229328767988]
  We introduce a unifying human-centered fairness framework that covers eight distinct fairness metrics.<n>Rather than privileging a single fairness notion, the framework enables stakeholders to assign weights across multiple fairness objectives.<n>We show that adjusting weights reveals nuanced trade-offs between different fairness metrics.
  arXiv  Detail & Related papers  (2025-12-07T17:52:38Z)
• Partial Identification Approach to Counterfactual Fairness Assessment [50.88100567472179]
  We introduce a Bayesian approach to bound unknown counterfactual fairness measures with high confidence.<n>Our results reveal a positive (spurious) effect on the COMPAS score when changing race to African-American (from all others) and a negative (direct causal) effect when transitioning from young to old age.
  arXiv  Detail & Related papers  (2025-09-30T18:35:08Z)
• Fairness-Aware Reinforcement Learning (FAReL): A Framework for Transparent and Balanced Sequential Decision-Making [41.53741129864172]
  Equity in real-world sequential decision problems can be enforced using fairness-aware methods.<n>We propose a framework where multiple trade-offs can be explored.<n>We show that our framework learns policies that are more fair across multiple scenarios, with only minor loss in performance reward.
  arXiv  Detail & Related papers  (2025-09-26T11:42:14Z)
• "I think this is fair'': Uncovering the Complexities of Stakeholder Decision-Making in AI Fairness Assessment [5.919313327612488]
  We show that stakeholders' fairness decisions are more complex than typical AI expert practices.<n>Our results extend the understanding of how stakeholders can meaningfully contribute to AI fairness governance and mitigation.
  arXiv  Detail & Related papers  (2025-09-22T16:12:12Z)
• EARN Fairness: Explaining, Asking, Reviewing, and Negotiating Artificial Intelligence Fairness Metrics Among Stakeholders [5.216732191267959]
  We propose a new framework, EARN Fairness, which facilitates collective metric decisions among stakeholders without requiring AI expertise.<n>The framework features an adaptable interactive system and a stakeholder-centered EARN Fairness process to explain fairness metrics, Ask stakeholders' personal metric preferences, Review metrics collectively, and Negotiate a consensus on metric selection.<n>Our work shows that the EARN Fairness framework enables stakeholders to express personal preferences and reach consensus, providing practical guidance for implementing human-centered AI fairness in high-risk contexts.
  arXiv  Detail & Related papers  (2024-07-16T07:20:30Z)
• The Impossibility of Fair LLMs [17.812295963158714]
  We analyze a variety of technical fairness frameworks and find inherent challenges in each that make the development of a fair language model intractable.<n>We show that each framework either does not extend to the general-purpose AI context or is infeasible in practice.<n>These inherent challenges would persist for general-purpose AI, including LLMs, even if empirical challenges, such as limited participatory input and limited measurement methods, were overcome.
  arXiv  Detail & Related papers  (2024-05-28T04:36:15Z)
• Towards Responsible AI in Banking: Addressing Bias for Fair Decision-Making [69.44075077934914]
  "Responsible AI" emphasizes the critical nature of addressing biases within the development of a corporate culture. This thesis is structured around three fundamental pillars: understanding bias, mitigating bias, and accounting for bias. In line with open-source principles, we have released Bias On Demand and FairView as accessible Python packages.
  arXiv  Detail & Related papers  (2024-01-13T14:07:09Z)
• Causal Fairness Analysis [68.12191782657437]
  We introduce a framework for understanding, modeling, and possibly solving issues of fairness in decision-making settings. The main insight of our approach will be to link the quantification of the disparities present on the observed data with the underlying, and often unobserved, collection of causal mechanisms. Our effort culminates in the Fairness Map, which is the first systematic attempt to organize and explain the relationship between different criteria found in the literature.
  arXiv  Detail & Related papers  (2022-07-23T01:06:34Z)
• Joint Multisided Exposure Fairness for Recommendation [76.75990595228666]
  This paper formalizes a family of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation.
  arXiv  Detail & Related papers  (2022-04-29T19:13:23Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Appropriate Fairness Perceptions? On the Effectiveness of Explanations in Enabling People to Assess the Fairness of Automated Decision Systems [0.0]
  We argue that for an effective explanation, perceptions of fairness should increase if and only if the underlying ADS is fair. In this in-progress work, we introduce the desideratum of appropriate fairness perceptions, propose a novel study design for evaluating it, and outline next steps towards a comprehensive experiment.
  arXiv  Detail & Related papers  (2021-08-14T09:39:59Z)
• Trustworthy AI [75.99046162669997]
  Brittleness to minor adversarial changes in the input data, ability to explain the decisions, address the bias in their training data, are some of the most prominent limitations. We propose the tutorial on Trustworthy AI to address six critical issues in enhancing user and public trust in AI systems.
  arXiv  Detail & Related papers  (2020-11-02T20:04:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.