FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation

• URL: http://arxiv.org/abs/2607.28934v1
• Date: Fri, 31 Jul 2026 01:29:16 GMT
• Title: FairFund-Bench: Evaluating Distributive Bias in LLM Resource Allocation
• Abstract summary: We introduce FairFund-Bench, a benchmark that systematically varies key features of previous audit designs.<n>We show that this disagreement can arise from differences in audit format and introduce FairFund-Bench.<n>The benchmark scores models on four criteria (demographic bias, deservingness alignment, cross-task consistency, and cross-context consistency) is publicly available and can be readily adapted to other substantive domains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on characteristics like race and gender. Recent LLM audits have produced inconsistent results, however, finding evidence of both positive and negative discrimination towards women and ethnic minorities, even for the same models. We show that this disagreement can arise from differences in audit format and introduce FairFund-Bench, a benchmark that systematically varies key features of previous audit designs: the evaluation task (rating, ranking, or allocation), comparison context (single or multi-stimulus), and whether the audit is transparent or disguised. The benchmark comprises 600 requests for financial assistance created from human-authored templates (calibrated against 1.3M real GoFundMe campaigns) across three domains, four race and two gender categories, and five causal framings of need derived from welfare deservingness theory. Across 14 models, audit format changes the direction of bias: models advantage minorities when rating claimants individually but penalize some groups when ranking them side by side. Bias magnitude, though small overall, is several times greater in disguised audits than in transparent ones, where, faced with appeals differing only in claimants' names, models overwhelmingly split funds equally. Causal framing effects, by contrast, exceed demographic effects by roughly an order of magnitude and are consistent across models and audit formats, indicating that current LLMs robustly reproduce human deservingness evaluations. The benchmark scores models on four criteria (demographic bias, deservingness alignment, cross-task consistency, and cross-context consistency), is publicly available, and can be readily adapted to other substantive domains.

Related papers

• The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits [2.347309464574674]
  A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side.<n>We test whether that reversal generalizes to hiring, lending, and medical triage.
  arXiv  Detail & Related papers  (2026-09-08T17:08:53Z)
• Do Fair Models Reason Fairly? Counterfactual Explanation Consistency for Procedural Fairness in Credit Decisions [0.0]
  We show that existing outcome-fair models can still apply fundamentally different reasoning to individuals.<n>We propose Counterfactual Explanation Consistency ( CEC), a framework that detects and mitigates this bias.<n>Key contributions include a nearest-neighbor counterfactual generation method, a modified baseline for integrated gradient comparisons, and a corresponding training loss.
  arXiv  Detail & Related papers  (2026-05-12T19:54:25Z)
• Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment [75.88231994139132]
  We show that even state-of-the-art models like GPT-4.1 achieve only 57.4% accuracy in predicting modal preferences.<n>We find significant pre-existing performance biases, where models better emulate young, male, Chinese, and Christian personas.
  arXiv  Detail & Related papers  (2026-04-14T15:06:13Z)
• Evaluating LLMs in Finance Requires Explicit Bias Consideration [88.38155218924999]
  Finance-specific biases can inflate performance, contaminate backtests, and make reported results useless for deployment claims.<n>No single bias is discussed in more than 28 percent of studies.<n>We propose a Structural Validity Framework and an evaluation checklist with minimal requirements for bias diagnosis and future system design.
  arXiv  Detail & Related papers  (2026-02-15T17:02:01Z)
• Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution [5.061421107401101]
  Large language models (LLMs) have achieved impressive performance, leading to their widespread adoption as decision-support tools in resource-constrained contexts like hiring and admissions.<n>There is, however, scientific consensus that AI systems can reflect and exacerbate societal biases, raising concerns about identity-based harm when used in critical social contexts.<n>In this work, we extend single-axis fairness evaluations to examine intersectional bias, recognizing that when multiple axes of discrimination intersect, they create distinct patterns of disadvantage.
  arXiv  Detail & Related papers  (2025-08-09T22:24:40Z)
• Robustly Improving LLM Fairness in Realistic Settings via Interpretability [0.16843915833103415]
  Anti-bias prompts fail when realistic contextual details are introduced.<n>We find that adding realistic context such as company names, culture descriptions from public careers pages, and selective hiring constraints induces significant racial and gender biases.<n>Our internal bias mitigation identifies race and gender-correlated directions and applies affine concept editing at inference time.
  arXiv  Detail & Related papers  (2025-06-12T17:34:38Z)
• Towards Large Language Models that Benefit for All: Benchmarking Group Fairness in Reward Models [16.977176752570617]
  Large Language Models (LLMs) are increasingly powerful and accessible to human users.<n> Ensuring fairness across diverse demographic groups, i.e., group fairness, is a critical ethical concern.<n>This work benchmarks the group fairness of learned reward models.
  arXiv  Detail & Related papers  (2025-03-10T19:39:39Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• Bias Similarity Measurement: A Black-Box Audit of Fairness Across LLMs [35.72288339965426]
  Bias Similarity Measurement (BSM) treats fairness as a relational property between models.<n>Open-weight models can match or exceed proprietary systems.<n>BSM offers an auditing workflow for procurement, regression testing, and lineage screening.
  arXiv  Detail & Related papers  (2024-10-15T19:21:14Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• What's in a Name? Auditing Large Language Models for Race and Gender Bias [45.1187517058961]
  We employ an audit design to investigate biases in state-of-the-art large language models, including GPT-4.<n>We find that the advice systematically disadvantages names that are commonly associated with racial minorities and women.
  arXiv  Detail & Related papers  (2024-02-21T18:25:25Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)
• Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models [73.24381010980606]
  This study examines issues of algorithmic fairness in the context of systems that inform tax audit selection by the IRS. We show how the use of more flexible machine learning methods for selecting audits may affect vertical equity. Our results have implications for the design of algorithmic tools across the public sector.
  arXiv  Detail & Related papers  (2022-06-20T16:27:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.