Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

• URL: http://arxiv.org/abs/2605.16996v1
• Date: Sat, 16 May 2026 13:44:06 GMT
• Title: Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
• Abstract summary: We induce personality in large language models by fine-tuning them on the long-form essays.<n>We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire.
• Score: 11.462572308067033
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.

Related papers

• Adaptive Interviewing for Persona Simulation in LLMs: Evidence-Grounded Reasoning Improves Decision Alignment [23.066564025282386]
  We propose an adaptive interview framework that gathers persona-relevant information through a structured three-stage dialogue.<n>We evaluate whether large language models can simulate participants' decisions in moral dilemma scenarios.
  arXiv  Detail & Related papers  (2026-05-28T06:53:08Z)
• Can LLMs Infer Personality from Real World Conversations? [5.705775078773656]
  Large Language Models (LLMs) offer a promising approach for scalable personality assessment from open-ended language.<n>Three state-of-the-art LLMs were tested using zero-shot prompting for BFI-10 item prediction and both zero-shot and chain-of-thought prompting for Big Five trait inference.<n>All models showed high test-retest reliability, but construct validity was limited.
  arXiv  Detail & Related papers  (2025-07-18T20:22:47Z)
• UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generation [69.13972417517233]
  Large Language Models (LLMs) are prone to hallucination, particularly in long-form generations.<n>We introduce UNCLE, a benchmark designed to evaluate uncertainty expression in both long- and short-form question answering (QA)<n>Our dataset is the first to directly link short- and long-form QA through aligned questions and gold-standard answers.
  arXiv  Detail & Related papers  (2025-05-22T17:16:08Z)
• Rediscovering the Latent Dimensions of Personality with Large Language Models as Trait Descriptors [4.814107439144414]
  We introduce a novel approach that uncovers latent personality dimensions in large language models (LLMs) Our experiments show that LLMs "rediscover" core personality traits such as extraversion, agreeableness, conscientiousness, neuroticism, and openness without relying on direct questionnaire inputs. We can use the derived principal components to assess personality along the Big Five dimensions, and achieve improvements in average personality prediction accuracy of up to 5% over fine-tuned models.
  arXiv  Detail & Related papers  (2024-09-16T00:24:40Z)
• LLMvsSmall Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model [58.887561071010985]
  Personality detection aims to detect one's personality traits underlying in social media posts. Most existing methods learn post features directly by fine-tuning the pre-trained language models. We propose a large language model (LLM) based text augmentation enhanced personality detection model.
  arXiv  Detail & Related papers  (2024-03-12T12:10:18Z)
• Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning [67.33899440998175]
  Chain-of-Opinion (COO) is a simple four-step solution modeling which and how to reason with personae.<n>COO distinguishes between explicit personae (demographics and ideology) and implicit personae (historical opinions)<n>COO efficiently achieves new state-of-the-art opinion prediction via prompting with only 5 inference calls, improving prior techniques by up to 4%.
  arXiv  Detail & Related papers  (2023-11-14T18:48:27Z)
• PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection [50.66968526809069]
  We propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection.
  arXiv  Detail & Related papers  (2023-10-31T08:23:33Z)
• Editing Personality for Large Language Models [73.59001811199823]
  This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs) We construct PersonalityEdit, a new benchmark dataset to address this task.
  arXiv  Detail & Related papers  (2023-10-03T16:02:36Z)
• Revisiting the Reliability of Psychological Scales on Large Language Models [62.57981196992073]
  This study aims to determine the reliability of applying personality assessments to Large Language Models. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory.
  arXiv  Detail & Related papers  (2023-05-31T15:03:28Z)
• PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits [30.770525830385637]
  We study the behavior of large language models (LLMs) based on the Big Five personality model. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types. Human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%.
  arXiv  Detail & Related papers  (2023-05-04T04:58:00Z)
• Can ChatGPT Assess Human Personalities? A General Evaluation Framework [70.90142717649785]
  Large Language Models (LLMs) have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests.
  arXiv  Detail & Related papers  (2023-03-01T06:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.