Personality Structured Interview for Large Language Model Simulation in Personality Research

• URL: http://arxiv.org/abs/2502.12109v1
• Date: Mon, 17 Feb 2025 18:31:57 GMT
• Title: Personality Structured Interview for Large Language Model Simulation in Personality Research
• Authors: Pengda Wang, Huiqi Zou, Hanjie Chen, Tianjun Sun, Ziang Xiao, Frederick L. Oswald,
• Abstract summary: We explore the potential of the theory-informed Personality Structured Interview as a tool for simulating human responses in personality research. We have provided a growing set of 357 structured interview transcripts from a representative sample, each containing an individual's response to 32 open-ended questions. Results from three experiments demonstrate that well-designed structured interviews could improve human-like heterogeneity in LLM-simulated personality data.
• Score: 8.208325358490807
• License:
• Abstract: Although psychometrics researchers have recently explored the use of large language models (LLMs) as proxies for human participants, LLMs often fail to generate heterogeneous data with human-like diversity, which diminishes their value in advancing social science research. To address these challenges, we explored the potential of the theory-informed Personality Structured Interview (PSI) as a tool for simulating human responses in personality research. In this approach, the simulation is grounded in nuanced real-human interview transcripts that target the personality construct of interest. We have provided a growing set of 357 structured interview transcripts from a representative sample, each containing an individual's response to 32 open-ended questions carefully designed to gather theory-based personality evidence. Additionally, grounded in psychometric research, we have summarized an evaluation framework to systematically validate LLM-generated psychometric data. Results from three experiments demonstrate that well-designed structured interviews could improve human-like heterogeneity in LLM-simulated personality data and predict personality-related behavioral outcomes (i.e., organizational citizenship behaviors and counterproductive work behavior). We further discuss the role of theory-informed structured interviews in LLM-based simulation and outline a general framework for designing structured interviews to simulate human-like data for psychometric research.

Related papers

• From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents [47.935533238820334]
  Traditional sociological research often relies on human participation, which, though effective, is expensive, challenging to scale, and with ethical concerns. Recent advancements in large language models (LLMs) highlight their potential to simulate human behavior, enabling the replication of individual responses and facilitating studies on many interdisciplinary studies. We categorize the simulations into three types: (1) Individual Simulation, which mimics specific individuals or demographic groups; (2) Scenario Simulation, where multiple agents collaborate to achieve goals within specific contexts; and (3) Simulation Society, which models interactions within agent societies to reflect the complexity and variety of real-world dynamics.
  arXiv  Detail & Related papers  (2024-12-04T18:56:37Z)
• Generative Agent Simulations of 1,000 People [56.82159813294894]
  We present a novel agent architecture that simulates the attitudes and behaviors of 1,052 real individuals. The generative agents replicate participants' responses on the General Social Survey 85% as accurately as participants replicate their own answers. Our architecture reduces accuracy biases across racial and ideological groups compared to agents given demographic descriptions.
  arXiv  Detail & Related papers  (2024-11-15T11:14:34Z)
• Designing LLM-Agents with Personalities: A Psychometric Approach [0.47498241053872914]
  This research introduces a novel methodology for assigning quantifiable, controllable and psychometrically validated personalities to Agents. It seeks to overcome the constraints of human subject studies, proposing Agents as an accessible tool for social science inquiry.
  arXiv  Detail & Related papers  (2024-10-25T01:05:04Z)
• 'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants [13.693069737188859]
  Recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development. We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift.
  arXiv  Detail & Related papers  (2024-09-28T18:28:47Z)
• PersLLM: A Personified Training Approach for Large Language Models [66.16513246245401]
  We propose PersLLM, integrating psychology-grounded principles of personality: social practice, consistency, and dynamic development. We incorporate personality traits directly into the model parameters, enhancing the model's resistance to induction, promoting consistency, and supporting the dynamic evolution of personality.
  arXiv  Detail & Related papers  (2024-07-17T08:13:22Z)
• Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models [57.518784855080334]
  Large Language Models (LLMs) have demonstrated exceptional task-solving capabilities, increasingly adopting roles akin to human-like assistants. This paper presents a framework for investigating psychology dimension in LLMs, including psychological identification, assessment dataset curation, and assessment with results validation. We introduce a comprehensive psychometrics benchmark for LLMs that covers six psychological dimensions: personality, values, emotion, theory of mind, motivation, and intelligence.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis [0.27309692684728604]
  We prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties.
  arXiv  Detail & Related papers  (2024-05-12T10:52:15Z)
• SeSaMe: A Framework to Simulate Self-Reported Ground Truth for Mental Health Sensing Studies [3.7398400615298466]
  Mental Models (SeSaMe) is a framework to alleviate participants' burden in digital mental health studies. By leveraging pre-trained large language models (LLMs), SeSaMe enables the simulation of participants' responses on psychological scales. We demonstrate an application of SeSaMe, where we use GPT-4 to simulate responses on one scale using responses from another as behavioral information.
  arXiv  Detail & Related papers  (2024-03-25T21:48:22Z)
• Human Simulacra: Benchmarking the Personification of Large Language Models [38.21708264569801]
  Large language models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This paper introduces a framework for constructing virtual characters' life stories from the ground up. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters.
  arXiv  Detail & Related papers  (2024-02-28T09:11:14Z)
• Editing Personality for Large Language Models [73.59001811199823]
  This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs) We construct PersonalityEdit, a new benchmark dataset to address this task.
  arXiv  Detail & Related papers  (2023-10-03T16:02:36Z)
• Revisiting the Reliability of Psychological Scales on Large Language Models [62.57981196992073]
  This study aims to determine the reliability of applying personality assessments to Large Language Models. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory.
  arXiv  Detail & Related papers  (2023-05-31T15:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.