Improving LLM Personas via Rationalization with Psychological Scaffolds

• URL: http://arxiv.org/abs/2504.17993v1
• Date: Fri, 25 Apr 2025 00:36:39 GMT
• Title: Improving LLM Personas via Rationalization with Psychological Scaffolds
• Authors: Brihi Joshi, Xiang Ren, Swabha Swayamdipta, Rik Koncel-Kedziorski, Tim Paek,
• Abstract summary: Language models prompted with a user description or persona can predict a user's preferences and opinions.<n>Existing approaches to building personas fail to capture the underlying reasoning behind said user judgments.<n>We introduce PB&J, a framework that improves LLM personas by incorporating rationales of why a user might make specific judgments.
• Score: 41.95479674995431
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models prompted with a user description or persona can predict a user's preferences and opinions, but existing approaches to building personas -- based solely on a user's demographic attributes and/or prior judgments -- fail to capture the underlying reasoning behind said user judgments. We introduce PB&J (Psychology of Behavior and Judgments), a framework that improves LLM personas by incorporating rationales of why a user might make specific judgments. These rationales are LLM-generated, and aim to reason about a user's behavior on the basis of their experiences, personality traits or beliefs. This is done using psychological scaffolds -- structured frameworks grounded in theories such as the Big 5 Personality Traits and Primal World Beliefs -- that help provide structure to the generated rationales. Experiments on public opinion and movie preference prediction tasks demonstrate that LLM personas augmented with PB&J rationales consistently outperform methods using only a user's demographics and/or judgments. Additionally, LLM personas constructed using scaffolds describing user beliefs perform competitively with those using human-written rationales.

Related papers

• Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models [50.16340812031201]
  We show that large language models (LLMs) do not update their beliefs as expected from the Bayesian framework.<n>We teach the LLMs to reason in a Bayesian manner by training them to mimic the predictions of an optimal Bayesian model.
  arXiv  Detail & Related papers  (2025-03-21T20:13:04Z)
• CBEval: A framework for evaluating and interpreting cognitive biases in LLMs [1.4633779950109127]
  Large Language models exhibit notable gaps in their cognitive processes. As reflections of human-generated data, these models have the potential to inherit cognitive biases.
  arXiv  Detail & Related papers  (2024-12-04T05:53:28Z)
• Large Language Models Assume People are More Rational than We Really are [10.857040292234984]
  In order for AI to communicate effectively with people, they must understand how we make decisions.<n>Previous empirical evidence seems to suggest that these implicit models are accurate.<n>We find that this is actually not the case when both simulating and predicting people's choices.
  arXiv  Detail & Related papers  (2024-06-24T18:15:27Z)
• Can LLM be a Personalized Judge? [24.858529542496367]
  We investigate the reliability of LLM-as-a-Personalized-Judge, asking LLMs to judge user preferences based on personas. Our findings suggest that directly applying LLM-as-a-Personalized-Judge is less reliable than previously assumed. We introduce verbal uncertainty estimation into the LLM-as-a-Personalized-Judge pipeline, allowing the model to express low confidence on uncertain judgments.
  arXiv  Detail & Related papers  (2024-06-17T15:41:30Z)
• PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models [25.657579792829743]
  We empirically evaluate how role-playing prompting influences Theory-of-Mind (ToM) reasoning capabilities. We propose the mechanism that, beyond the inherent variance in the complexity of reasoning tasks, performance differences arise because of socially-motivated prompting differences.
  arXiv  Detail & Related papers  (2024-03-04T17:34:34Z)
• On the steerability of large language models toward data-driven personas [98.9138902560793]
  Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented. Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs.
  arXiv  Detail & Related papers  (2023-11-08T19:01:13Z)
• MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks [49.60689355674541]
  A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments. We test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with human participants.
  arXiv  Detail & Related papers  (2023-10-30T15:57:32Z)
• Character-LLM: A Trainable Agent for Role-Playing [67.35139167985008]
  Large language models (LLMs) can be used to serve as agents to simulate human behaviors. We introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc.
  arXiv  Detail & Related papers  (2023-10-16T07:58:56Z)
• Aligning Language Models to User Opinions [10.953326025836475]
  We find that the opinions of a user and their demographics and ideologies are not mutual predictors. We use this insight to align LLMs by modeling both user opinions as well as user demographics and ideology. In addition to the typical approach of prompting LLMs with demographics and ideology, we discover that utilizing the most relevant past opinions from individual users enables the model to predict user opinions more accurately.
  arXiv  Detail & Related papers  (2023-05-24T09:11:11Z)
• Can ChatGPT Assess Human Personalities? A General Evaluation Framework [70.90142717649785]
  Large Language Models (LLMs) have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests.
  arXiv  Detail & Related papers  (2023-03-01T06:16:14Z)
• A Neural Topical Expansion Framework for Unstructured Persona-oriented Dialogue Generation [52.743311026230714]
  Persona Exploration and Exploitation (PEE) is able to extend the predefined user persona description with semantically correlated content. PEE consists of two main modules: persona exploration and persona exploitation. Our approach outperforms state-of-the-art baselines in terms of both automatic and human evaluations.
  arXiv  Detail & Related papers  (2020-02-06T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.