SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models

• URL: http://arxiv.org/abs/2604.02660v1
• Date: Fri, 03 Apr 2026 02:47:37 GMT
• Title: SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models
• Abstract summary: We introduce SocioEval, a template-based framework for evaluating socioeconomic bias in foundation models through decision-making tasks.<n>Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$times$ higher bias than education-related decisions.<n>SocioEval provides a scalable, foundation for auditing class-based bias in language models.
• Score: 0.19703625025720697
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As Large Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240 prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42\%-33.75\%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$\times$ higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for auditing class-based bias in language models.

Related papers

• To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias [61.40435736418359]
  We introduce a unified framework that standardizes heterogeneous benchmarks to contrast isolated demographic assessments with forced-choice comparative settings.<n>We show that comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts.<n>Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size.
  arXiv  Detail & Related papers  (2026-06-23T13:53:50Z)
• Sociodemographic Biases in Educational Counselling by Large Language Models [61.30199414932204]
  This study examines sociodemographic biases in Large Language Models (LLMs)<n>We evaluate responses from six LLMs answering questions about 900 vignettes describing students in diverse circumstances.
  arXiv  Detail & Related papers  (2026-04-03T14:08:22Z)
• The Bias is in the Details: An Assessment of Cognitive Bias in LLMs [1.9651471153360769]
  Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes.<n>This paper presents a large-scale evaluation of eight well-established cognitive biases across 45 LLMs.<n>We find that both model size and prompt specificity play a significant role on bias susceptibility.
  arXiv  Detail & Related papers  (2025-09-26T19:09:42Z)
• Intrinsic Meets Extrinsic Fairness: Assessing the Downstream Impact of Bias Mitigation in Large Language Models [11.396244643030983]
  Large Language Models (LLMs) exhibit socio-economic biases that can propagate into downstream tasks.<n>We present a unified evaluation framework to compare intrinsic bias mitigation via concept unlearning with extrinsic bias mitigation via counterfactual data augmentation.<n>Our results show that intrinsic bias mitigation through unlearning reduces intrinsic gender bias by up to 94.9%, while also improving downstream task fairness metrics, such as demographic parity by up to 82%, without compromising accuracy.
  arXiv  Detail & Related papers  (2025-09-19T22:59:55Z)
• MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs via Theory of Mind [27.209638457499427]
  Theory of Mind (ToM) in Large Language Models (LLMs) refers to their capacity for reasoning about mental states.<n>We propose an evaluation framework that leverages the Stereotype Content Model (SCM) to reconceptualize bias as a multi-dimensional failure in ToM across Competence, Sociability, and Morality.
  arXiv  Detail & Related papers  (2025-06-17T03:50:57Z)
• The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models [91.86718720024825]
  We center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias.<n>Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning.<n>We conclude with recommendations tailored to DPO and broader alignment practices.
  arXiv  Detail & Related papers  (2024-11-06T06:50:50Z)
• Investigating Implicit Bias in Large Language Models: A Large-Scale Study of Over 50 LLMs [0.0]
  Large Language Models (LLMs) are being adopted across a wide range of tasks. Recent research indicates that LLMs can harbor implicit biases even when they pass explicit bias evaluations. This study highlights that newer or larger language models do not automatically exhibit reduced bias.
  arXiv  Detail & Related papers  (2024-10-13T03:43:18Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model [72.13121434085116]
  We introduce VLBiasBench, a benchmark to evaluate biases in Large Vision-Language Models (LVLMs)<n>VLBiasBench features a dataset that covers nine distinct categories of social biases, including age, disability status, gender, nationality, physical appearance, race, religion, profession, social economic status, as well as two intersectional bias categories: race x gender and race x social economic status.<n>We conduct extensive evaluations on 15 open-source models as well as two advanced closed-source models, yielding new insights into the biases present in these models.
  arXiv  Detail & Related papers  (2024-06-20T10:56:59Z)
• Understanding Intrinsic Socioeconomic Biases in Large Language Models [4.276697874428501]
  We introduce a novel dataset of one million English sentences to quantify socioeconomic biases. Our findings reveal pervasive socioeconomic biases in both established models like GPT-2 and state-of-the-art models like Llama 2 and Falcon.
  arXiv  Detail & Related papers  (2024-05-28T23:54:44Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.