What Limits LLM-based Human Simulation: LLMs or Our Design?

• URL: http://arxiv.org/abs/2501.08579v1
• Date: Wed, 15 Jan 2025 04:59:49 GMT
• Title: What Limits LLM-based Human Simulation: LLMs or Our Design?
• Authors: Qian Wang, Jiaying Wu, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, Bingsheng He,
• Abstract summary: We argue that advancing LLM-based human simulation requires addressing both LLM's inherent limitations and simulation framework design challenges. To support further research in this field, we provide a curated collection of LLM-based human simulation resources.
• Score: 43.54008528543768
• License:
• Abstract: We argue that advancing LLM-based human simulation requires addressing both LLM's inherent limitations and simulation framework design challenges. Recent studies have revealed significant gaps between LLM-based human simulations and real-world observations, highlighting these dual challenges. To address these gaps, we present a comprehensive analysis of LLM limitations and our design issues, proposing targeted solutions for both aspects. Furthermore, we explore future directions that address both challenges simultaneously, particularly in data collection, LLM generation, and evaluation. To support further research in this field, we provide a curated collection of LLM-based human simulation resources.\footnote{https://github.com/Persdre/llm-human-simulation}

Related papers

• LLM-Mirror: A Generated-Persona Approach for Survey Pre-Testing [0.0]
  We investigate whether providing respondents' prior information can replicate both statistical distributions and individual decision-making patterns. We also introduce the concept of the LLM-Mirror, user personas generated by supplying respondent-specific information to the LLM. Our findings show that: (1) PLS-SEM analysis shows LLM-generated responses align with human responses, (2) LLMs are capable of reproducing individual human responses, and (3) LLM-Mirror responses closely follow human responses at the individual level.
  arXiv  Detail & Related papers  (2024-12-04T09:39:56Z)
• Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations? [40.00556764679785]
  Large Language Models (LLMs) are increasingly employed for simulations, enabling applications in role-playing agents and Computational Social Science (CSS) In this paper, we aim to answer How reliable is LLM-based simulation?''
  arXiv  Detail & Related papers  (2024-10-30T20:09:37Z)
• From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future [15.568939568441317]
  We investigate the current practice and solutions for large language models (LLMs) and LLM-based agents for software engineering. In particular we summarise six key topics: requirement engineering, code generation, autonomous decision-making, software design, test generation, and software maintenance. We discuss the models and benchmarks used, providing a comprehensive analysis of their applications and effectiveness in software engineering.
  arXiv  Detail & Related papers  (2024-08-05T14:01:15Z)
• Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning [53.6472920229013]
  Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. LLMs are prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. We introduce Q*, a framework for guiding LLMs decoding process with deliberative planning.
  arXiv  Detail & Related papers  (2024-06-20T13:08:09Z)
• Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks. The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
  arXiv  Detail & Related papers  (2024-01-30T17:38:54Z)
• CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations [61.9212914612875]
  We present a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic. We use this framework to measure open-ended LLM simulations' susceptibility to caricature, defined via two criteria: individuation and exaggeration. We find that for GPT-4, simulations of certain demographics (political and marginalized groups) and topics (general, uncontroversial) are highly susceptible to caricature.
  arXiv  Detail & Related papers  (2023-10-17T18:00:25Z)
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity [61.54815512469125]
  This survey addresses the crucial issue of factuality in Large Language Models (LLMs) As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
  arXiv  Detail & Related papers  (2023-10-11T14:18:03Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.