Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs

• URL: http://arxiv.org/abs/2506.02659v1
• Date: Tue, 03 Jun 2025 09:12:23 GMT
• Title: Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
• Authors: Manon Reusens, Bart Baesens, David Jurgens,
• Abstract summary: We introduce a new standardized framework to analyze consistency in persona-assigned Large Language Models (LLMs)<n>Our framework evaluates personas across four different categories (happiness, occupation, personality, and political stance) spanning multiple task dimensions.<n>Our findings reveal that consistency is influenced by multiple factors, including the assigned persona, stereotypes, and model design choices.
• Score: 12.780044838203738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized Large Language Models (LLMs) are increasingly used in diverse applications, where they are assigned a specific persona - such as a happy high school teacher - to guide their responses. While prior research has examined how well LLMs adhere to predefined personas in writing style, a comprehensive analysis of consistency across different personas and task types is lacking. In this paper, we introduce a new standardized framework to analyze consistency in persona-assigned LLMs. We define consistency as the extent to which a model maintains coherent responses when assigned the same persona across different tasks and runs. Our framework evaluates personas across four different categories (happiness, occupation, personality, and political stance) spanning multiple task dimensions (survey writing, essay generation, social media post generation, single turn, and multi-turn conversations). Our findings reveal that consistency is influenced by multiple factors, including the assigned persona, stereotypes, and model design choices. Consistency also varies across tasks, increasing with more structured tasks and additional context. All code is available on GitHub.

Related papers

• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations [112.81207927088117]
  PersonaConvBench is a benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs)<n>We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements.
  arXiv  Detail & Related papers  (2025-05-20T09:13:22Z)
• A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs [21.08821957575833]
  We create a novel dataset of 40 diverse personas varying in age, gender, occupation, and location.<n>This consists of four modalities to equivalently represent a persona: image-only, text-only, a combination of image and small text, and typographical images.<n> Comprehensive experiments show that personas represented by detailed text show more linguistic habits, while typographical images often show more consistency with the persona.
  arXiv  Detail & Related papers  (2025-02-27T20:25:00Z)
• Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMs [50.0874045899661]
  We introduce CharacterBot, a model designed to replicate both the linguistic patterns and distinctive thought patterns as manifested in the textual works of a character.<n>Using Lu Xun, a renowned Chinese writer as a case study, we propose four training tasks derived from his 17 essay collections.<n>These include a pre-training task focused on mastering external linguistic structures and knowledge, as well as three fine-tuning tasks.<n>We evaluate CharacterBot on three tasks for linguistic accuracy and opinion comprehension, demonstrating that it significantly outperforms the baselines on our adapted metrics.
  arXiv  Detail & Related papers  (2025-02-18T16:11:54Z)
• Can LLM Agents Maintain a Persona in Discourse? [3.286711575862228]
  Large Language Models (LLMs) are widely used as conversational agents, exploiting their capabilities in various sectors such as education, law, medicine, and more.<n>LLMs are often subjected to context-shifting behaviour, resulting in a lack of consistent and interpretable personality-aligned interactions.<n>We show that while LLMs can be guided toward personality-driven dialogue, their ability to maintain personality traits varies significantly depending on the combination of models and discourse settings.
  arXiv  Detail & Related papers  (2025-02-17T14:36:39Z)
• Aligning LLMs with Individual Preferences via Interaction [51.72200436159636]
  We train large language models (LLMs) that can ''interact to align''<n>We develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures.<n>For evaluation, we establish the ALOE benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations.
  arXiv  Detail & Related papers  (2024-10-04T17:48:29Z)
• PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection [50.66968526809069]
  We propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner. Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection.
  arXiv  Detail & Related papers  (2023-10-31T08:23:33Z)
• Editing Personality for Large Language Models [73.59001811199823]
  This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs) We construct PersonalityEdit, a new benchmark dataset to address this task.
  arXiv  Detail & Related papers  (2023-10-03T16:02:36Z)
• Teach LLMs to Personalize -- An Approach inspired by Writing Education [37.198598706659524]
  We propose a general approach for personalized text generation using large language models (LLMs) Inspired by the practice of writing education, we develop a multistage and multitask framework to teach LLMs for personalized generation.
  arXiv  Detail & Related papers  (2023-08-15T18:06:23Z)
• Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation [85.32991360774447]
  Natural language generation (NLG) spans a broad range of tasks, each of which serves for specific objectives. We propose a unifying perspective based on the nature of information change in NLG tasks. We develop a family of interpretable metrics that are suitable for evaluating key aspects of different NLG tasks.
  arXiv  Detail & Related papers  (2021-09-14T01:00:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.