The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models

• URL: http://arxiv.org/abs/2507.16076v1
• Date: Mon, 21 Jul 2025 21:23:29 GMT
• Title: The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models
• Authors: Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, Markus Strohmaier,
• Abstract summary: We show how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence large language models.<n>Our findings show that LLMs struggle to simulate marginalized groups, particularly nonbinary, Hispanic, and Middle Eastern identities.<n>Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment.
• Score: 3.2919397230854983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Persona prompting is increasingly used in large language models (LLMs) to simulate views of various sociodemographic groups. However, how a persona prompt is formulated can significantly affect outcomes, raising concerns about the fidelity of such simulations. Using five open-source LLMs, we systematically examine how different persona prompt strategies, specifically role adoption formats and demographic priming strategies, influence LLM simulations across 15 intersectional demographic groups in both open- and closed-ended tasks. Our findings show that LLMs struggle to simulate marginalized groups, particularly nonbinary, Hispanic, and Middle Eastern identities, but that the choice of demographic priming and role adoption strategy significantly impacts their portrayal. Specifically, we find that prompting in an interview-style format and name-based priming can help reduce stereotyping and improve alignment. Surprisingly, smaller models like OLMo-2-7B outperform larger ones such as Llama-3.3-70B. Our findings offer actionable guidance for designing sociodemographic persona prompts in LLM-based simulation studies.

Related papers

• Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment [49.81946749379338]
  This work seeks to analyze the capacity of Transformers-based systems to learn demographic biases present in the data.<n>We propose a privacy-enhancing framework to reduce gender information from the learning pipeline as a way to mitigate biased behaviors in the final tools.
  arXiv  Detail & Related papers  (2025-06-13T15:29:43Z)
• Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalization [6.781972039785424]
  Generative Large Language Models (LLMs) infer user's demographic information from subtle cues in the conversation.<n>Our results highlight the need for greater transparency and control in how LLMs represent user identity.
  arXiv  Detail & Related papers  (2025-05-22T09:48:51Z)
• Deep Binding of Language Model Virtual Personas: a Study on Approximating Political Partisan Misperceptions [4.234771450043289]
  Large language models (LLMs) are increasingly capable of simulating human behavior.<n>We propose a novel methodology for constructing virtual personas with synthetic user backstories" generated as extended, multi-turn interview transcripts.<n>Our generated backstories are longer, rich in detail, and consistent in authentically describing a singular individual.
  arXiv  Detail & Related papers  (2025-04-16T00:10:34Z)
• Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions [33.76973308687867]
  We show that models do improve in sociodemographic prompting when trained.<n>This performance gain is largely due to models learning annotator-specific behaviour rather than sociodemographic patterns.<n>Across all tasks, our results suggest that models learn little meaningful connection between sociodemographics and annotation.
  arXiv  Detail & Related papers  (2025-02-28T09:53:42Z)
• Actions Speak Louder than Words: Agent Decisions Reveal Implicit Biases in Language Models [10.565316815513235]
  Large language models (LLMs) may still exhibit implicit biases when simulating human behavior.<n>We show that state-of-the-art LLMs exhibit significant sociodemographic disparities in nearly all simulations.<n>When comparing our findings to real-world disparities reported in empirical studies, we find that the biases we uncovered are directionally aligned but markedly amplified.
  arXiv  Detail & Related papers  (2025-01-29T05:21:31Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• Hate Personified: Investigating the role of LLMs in content moderation [64.26243779985393]
  For subjective tasks such as hate detection, where people perceive hate differently, the Large Language Model's (LLM) ability to represent diverse groups is unclear. By including additional context in prompts, we analyze LLM's sensitivity to geographical priming, persona attributes, and numerical information to assess how well the needs of various groups are reflected.
  arXiv  Detail & Related papers  (2024-10-03T16:43:17Z)
• Social Debiasing for Fair Multi-modal LLMs [55.8071045346024]
  Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC) and ii) Proposing an Anti-Stereotype Debiasing strategy (ASD)
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptive [53.08398658452411]
  Large Language Models (LLMs) are increasingly utilized in autonomous decision-making.<n>We show that this sampling behavior resembles that of human decision-making.<n>We show that this deviation of a sample from the statistical norm towards a prescriptive component consistently appears in concepts across diverse real-world domains.
  arXiv  Detail & Related papers  (2024-02-16T18:28:43Z)
• Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs [13.744746481528711]
  Large Language Models (LLMs) are widely used to simulate human responses across diverse contexts.<n>We evaluate nine popular LLMs on their ability to understand demographic differences in two subjective judgment tasks: politeness and offensiveness.<n>We find that in zero-shot settings, most models' predictions for both tasks align more closely with labels from White participants than those from Asian or Black participants.
  arXiv  Detail & Related papers  (2023-11-16T10:02:24Z)
• On the steerability of large language models toward data-driven personas [98.9138902560793]
  Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs.
  arXiv  Detail & Related papers  (2023-11-08T19:01:13Z)
• Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting [64.80538055623842]
  sociodemographic prompting is a technique that steers the output of prompt-based models towards answers that humans with specific sociodemographic profiles would give. We show that sociodemographic information affects model predictions and can be beneficial for improving zero-shot learning in subjective NLP tasks.
  arXiv  Detail & Related papers  (2023-09-13T15:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.