Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

• URL: http://arxiv.org/abs/2606.12730v1
• Date: Wed, 10 Jun 2026 22:28:53 GMT
• Title: Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior
• Abstract summary: We run experiments across four behavioral tasks and 11 frontier LLMs.<n>We find that SR-behavior coherence exists but is selective.<n>These findings suggest that coarse personality frameworks may not be the best tools for testing deployment behavior.
• Score: 58.538864650160406
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anticipating LLM behavioral tendencies from low-cost psychometric probes is critical for safe deployment, but only if self-reports (SR) reliably predict behavior. Recent work documented substantial SR-behavior dissociation in LLMs, but relied on broad personality traits (Big 5) that predict specific behaviors weakly, even in humans. Furthermore, the isolation of conversational sessions combined with weak context matching left open whether LLMs truly lack coherence or whether the conditions needed to detect such coherence were not met. We contrast Big 5 with the Theory of Planned Behavior (TPB), which measures intention targeted to a specific behavior and predicts human behavior substantially better than broad traits. We run experiments across four behavioral tasks and 11 frontier LLMs, while also varying session context and identity induction. We find that SR-behavior coherence exists but is selective. 1) Within a shared conversation, the Theory of Planned Behavior reaches human-level coherence; Big 5 does not. 2) Across separate conversations, coherence survives only for behaviors anchored outside the immediate prompt, such as implicit bias shaped by training, and collapses when behavior is strongly primed by context, as with sycophancy. 3) Persona prompting makes self-reports more consistent across conversations, but does not bring behavior into alignment. These findings suggest that coarse personality frameworks, such as Big 5 may not be the best tools for testing deployment behavior. More task- and behavior-specific instruments are needed, and even these must be evaluated across tasks and contexts.

Related papers

• DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents [67.07361089429757]
  We measure real-time floor adherence using six real-time speech systems.<n>We find architecture-dependent trade-offs.<n>GPT-Realtime, Mini-o Duplex, Fun-Audio-Chat strongly adhere to persona-consistent content.
  arXiv  Detail & Related papers  (2026-09-03T06:28:02Z)
• Your LLM, Your Style: Behavioral Mode Axes for LLM Behavioral Control [15.236237731175192]
  Large language models (LLMs) increasingly act in interactive settings where their behavioral styles affect user experience, safety, and downstream decision making.<n>Existing LLM personality studies largely rely on self-report questionnaires administered in first-person settings.<n>We introduce a situated behavioral-data framework for studying and controlling LLM behavioral personality.
  arXiv  Detail & Related papers  (2026-08-11T09:25:47Z)
• Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events [34.88005801095875]
  Personality-conditioned LLM agents (PC-Agents) are increasingly used in emotional support, social simulation, and role-playing.<n>We study event-induced personality change after 11 major life events using the Big Five traits as a psychometric anchor.<n>Our results suggest that current PC-Agents simulate the mean of human personality dynamics, but not its shape.
  arXiv  Detail & Related papers  (2026-08-06T18:25:58Z)
• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• From Representations to Behaviors: Exploring the Person-Situation-Behavior Triad in LLMs [8.188989044347595]
  Existing studies of personality-related behavior in LLMs have primarily focused on outputs elicited under personality conditioning.<n>We introduce a framework for discovering, controlling, and validating trait-like representations in LLMs.<n>Our findings provide evidence that LLMs contain controllable trait-like representations linking internal states, situational expression, and behavioral outcomes.
  arXiv  Detail & Related papers  (2026-07-29T12:36:47Z)
• Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction [21.123771789383323]
  Simulating human conversations using large language models (LLMs) has emerged as a scalable methodology for modeling human social interaction.<n>We introduce CoCoEval, an evaluation framework that analyzes LLM-simulated conversations by detecting 10 types of inconsistent and uncollaborative behaviors.<n>Our analysis shows that under vanilla prompting, LLM-simulated conversations exhibit far fewer inconsistent and uncollaborative behaviors than human conversations.
  arXiv  Detail & Related papers  (2026-03-17T19:29:50Z)
• Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution [7.599497643290519]
  Large language models (LLMs) are increasingly used to simulate human behavior in social settings.<n>It remains unclear whether these simulations reproduce the personality-behavior patterns observed in humans.
  arXiv  Detail & Related papers  (2026-02-07T07:20:24Z)
• Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process [66.38541693477181]
  We propose an unsupervised framework for discovering reasoning vectors, which we define as directions in the activation space that encode distinct reasoning behaviors.<n>By segmenting chain-of-thought traces into sentence-level'steps', we uncover disentangled features corresponding to interpretable behaviors such as reflection and backtracking.<n>We demonstrate the ability to control response confidence by identifying confidence-related vectors in the SAE decoder space.
  arXiv  Detail & Related papers  (2025-12-30T05:09:11Z)
• The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs [60.15472325639723]
  Personality traits have long been studied as predictors of human behavior.<n>Recent advances in Large Language Models (LLMs) suggest similar patterns may emerge in artificial systems.
  arXiv  Detail & Related papers  (2025-09-03T21:27:10Z)
• Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation History [7.58175460763641]
  Even 400B+ models exhibit substantial response variability.<n> Interventions expected to stabilize behavior, such as chain-of-thought reasoning, detailed personas instruction, inclusion of conversation history, can paradoxically increase variability.<n>For safety-critical applications requiring predictable behavior, these findings indicate that personality-based alignment strategies may be fundamentally inadequate.
  arXiv  Detail & Related papers  (2025-08-06T19:11:33Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Revisiting the Reliability of Psychological Scales on Large Language Models [62.57981196992073]
  This study aims to determine the reliability of applying personality assessments to Large Language Models. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory.
  arXiv  Detail & Related papers  (2023-05-31T15:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.