Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference

• URL: http://arxiv.org/abs/2608.26674v1
• Date: Thu, 27 Aug 2026 06:27:51 GMT
• Title: Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
• Abstract summary: PRISM (Persona Reasoning with Inverse SFL-based Modeling) is a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task.<n>Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style.<n>Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.
• Score: 21.264274413987575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models are increasingly deployed to simulate diverse human characters, ensuring persona fidelity, defined as the extent to which an agent's behavior consistently reflects the psychological and stylistic characteristics of a target persona, has become a critical requirement. However, existing evaluation paradigms primarily rely on either holistic LLM-based judges, which are prone to "holistic appraisal hallucination'', or static psychometric inventories, which fail to capture the context-dependent fidelity required in dynamic dialogue. To address these limitations, we propose PRISM (Persona Reasoning with Inverse SFL-based Modeling), a psycholinguistically grounded framework that reformulates persona fidelity evaluation as a structured inverse inference task. Inspired by Systemic Functional Linguistics (SFL), PRISM decomposes persona fidelity into three functional dimensions: Task Framing, Interpersonal Stance, and Linguistic Style. It estimates dimension-specific evidence over a persona-conditioned label space and aggregates these signals into an interpretable and auditable evaluation process. Experiments show that PRISM yields more accurate and stable judgements than traditional holistic judging, providing a more reliable framework for persona fidelity evaluation.

Related papers

• Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models [49.45276299939287]
  Large Language Models (LLMs) increasingly handle complex subjective tasks.<n>LLMs fluctuate unpredictably under minor wording changes.<n> resolving this duality is critical for reliable AI alignment.
  arXiv  Detail & Related papers  (2026-09-15T03:35:04Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• Eval4Sim: An Evaluation Framework for Persona Simulation [9.555170143860694]
  Eval4Sim is an evaluation framework that measures how closely simulated conversations align with human conversational patterns.<n>Unlike absolute or optimization-oriented metrics, Eval4Sim uses a human conversational corpus as a reference baseline and penalizes deviations in both directions.
  arXiv  Detail & Related papers  (2026-03-03T11:30:50Z)
• From Classification to Ranking: Enhancing LLM Reasoning Capabilities for MBTI Personality Detection [25.825456002235967]
  Personality detection aims to measure an individual's corresponding personality traits through their social media posts.<n>Existing approaches enhance personality trait analysis by leveraging Large Language Models (LLMs)<n>We propose a corresponding reinforcement learning training paradigm for personality detection.
  arXiv  Detail & Related papers  (2026-01-26T15:28:43Z)
• Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions [11.415343473837583]
  Persona-assigned large language models (LLMs) are used in domains such as education, healthcare, and sociodemographic simulation.<n>We introduce an evaluation protocol that combines long persona dialogues and evaluation datasets to create dialogue-conditioned benchmarks.<n>We find that persona fidelity degrades over the course of dialogues, especially in goal-oriented conversations.
  arXiv  Detail & Related papers  (2025-12-14T17:27:02Z)
• TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation [55.55404595177229]
  Large Language Models (LLMs) are exhibiting emergent human-like abilities.<n>TwinVoice is a benchmark for assessing persona simulation across diverse real-world contexts.
  arXiv  Detail & Related papers  (2025-10-29T14:00:42Z)
• Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation [16.76995815742803]
  We propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity.<n>Our three key metrics measure the degree of persona alignment and consistency within and across generations.<n>By analyzing persona fidelity across diverse tasks and personality types, we reveal how task structure and persona desirability influence model adaptability.
  arXiv  Detail & Related papers  (2025-06-24T06:33:10Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models [53.00812898384698]
  We argue that human evaluation of generative large language models (LLMs) should be a multidisciplinary undertaking. We highlight how cognitive biases can conflate fluent information and truthfulness, and how cognitive uncertainty affects the reliability of rating scores such as Likert. We propose the ConSiDERS-The-Human evaluation framework consisting of 6 pillars -- Consistency, Scoring Criteria, Differentiating, User Experience, Responsible, and Scalability.
  arXiv  Detail & Related papers  (2024-05-28T22:45:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.