Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

• URL: http://arxiv.org/abs/2505.16164v1
• Date: Thu, 22 May 2025 03:08:27 GMT
• Title: Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task
• Authors: Mengyang Qiu, Zoe Brisebois, Siena Sun,
• Abstract summary: Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks.<n>This study examines whether LLMs can approximate individual differences in the phonemic fluency task.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 model configurations, varying prompt specificity, sampling temperature, and model type, and compared outputs to responses from 106 human participants. While some configurations, especially Claude 3.7 Sonnet, matched human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse and structurally rigid, and LLM ensembles failed to increase diversity. Network analyses further revealed fundamental differences in retrieval structure between humans and models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.

Related papers

• Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? [84.32752330104775]
  Variation in human annotation (i.e., disagreements) is common in NLP.<n>We evaluate the influence of different reasoning settings on Large Language Model disagreement modeling.<n>Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling.
  arXiv  Detail & Related papers  (2025-06-24T09:49:26Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns [0.0]
  We investigate the choice patterns of Large Language Models (LLMs) in the context of Decisions from Experience tasks.<n>We find that on the aggregate, LLMs appear to display behavioral biases similar to humans.<n>However, more nuanced analyses of the choice patterns reveal that this happens for very different reasons.
  arXiv  Detail & Related papers  (2025-03-13T10:47:03Z)
• One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity [2.5975241792179378]
  Researchers have proposed using large language models (LLMs) as replacements for humans in behavioral research. It is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. We use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability.
  arXiv  Detail & Related papers  (2024-11-07T04:38:58Z)
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? [60.01746782465275]
  Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.
  arXiv  Detail & Related papers  (2024-10-07T02:30:18Z)
• HLB: Benchmarking LLMs' Humanlikeness in Language Use [2.438748974410787]
  We present a comprehensive humanlikeness benchmark (HLB) evaluating 20 large language models (LLMs) We collected responses from over 2,000 human participants and compared them to outputs from the LLMs in these experiments. Our results reveal fine-grained differences in how well LLMs replicate human responses across various linguistic levels.
  arXiv  Detail & Related papers  (2024-09-24T09:02:28Z)
• Cognitive phantoms in LLMs through the lens of latent variables [0.3441021278275805]
  Large language models (LLMs) increasingly reach real-world applications, necessitating a better understanding of their behaviour. Recent studies administering psychometric questionnaires to LLMs report human-like traits in LLMs, potentially influencing behaviour. This approach suffers from a validity problem: it presupposes that these traits exist in LLMs and that they are measurable with tools designed for humans. This study investigates this problem by comparing latent structures of personality between humans and three LLMs using two validated personality questionnaires.
  arXiv  Detail & Related papers  (2024-09-06T12:42:35Z)
• Explaining Large Language Models Decisions Using Shapley Values [1.223779595809275]
  Large language models (LLMs) have opened up exciting possibilities for simulating human behavior and cognitive processes. However, the validity of utilizing LLMs as stand-ins for human subjects remains uncertain. This paper presents a novel approach based on Shapley values to interpret LLM behavior and quantify the relative contribution of each prompt component to the model's output.
  arXiv  Detail & Related papers  (2024-03-29T22:49:43Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• Revisiting the Reliability of Psychological Scales on Large Language Models [62.57981196992073]
  This study aims to determine the reliability of applying personality assessments to Large Language Models. Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory.
  arXiv  Detail & Related papers  (2023-05-31T15:03:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.