Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism

• URL: http://arxiv.org/abs/2506.21974v1
• Date: Fri, 27 Jun 2025 07:32:16 GMT
• Title: Don't Trust Generative Agents to Mimic Communication on Social Networks Unless You Benchmarked their Empirical Realism
• Authors: Simon Münker, Nils Schwager, Achim Rettinger,
• Abstract summary: We focus on replicating the behavior of social network users with the use of Large Language Models.<n>We empirically test different approaches to imitate user behavior on X in English and German.<n>Our findings suggest that social simulations should be validated by their empirical realism measured in the setting in which the simulation components were fitted.
• Score: 1.734165485480267
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability of Large Language Models (LLMs) to mimic human behavior triggered a plethora of computational social science research, assuming that empirical studies of humans can be conducted with AI agents instead. Since there have been conflicting research findings on whether and when this hypothesis holds, there is a need to better understand the differences in their experimental designs. We focus on replicating the behavior of social network users with the use of LLMs for the analysis of communication on social networks. First, we provide a formal framework for the simulation of social networks, before focusing on the sub-task of imitating user communication. We empirically test different approaches to imitate user behavior on X in English and German. Our findings suggest that social simulations should be validated by their empirical realism measured in the setting in which the simulation components were fitted. With this paper, we argue for more rigor when applying generative-agent-based modeling for social simulation.

Related papers

• DynamiX: Large-Scale Dynamic Social Network Simulator [101.65679342680542]
  DynamiX is a novel large-scale social network simulator dedicated to dynamic social network modeling.<n>For opinion leaders, we propose an information-stream-based link prediction method recommending potential users with similar stances.<n>For ordinary users, we construct an inequality-oriented behavior decision-making module.
  arXiv  Detail & Related papers  (2025-07-26T12:13:30Z)
• YuLan-OneSim: Towards the Next Generation of Social Simulator with Large Language Models [50.86336063222539]
  We introduce a novel social simulator called YuLan-OneSim.<n>Users can simply describe and refine their simulation scenarios through natural language interactions with our simulator.<n>We implement 50 default simulation scenarios spanning 8 domains, including economics, sociology, politics, psychology, organization, demographics, law, and communication.
  arXiv  Detail & Related papers  (2025-05-12T14:05:17Z)
• SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Users [70.02370111025617]
  We introduce SocioVerse, an agent-driven world model for social simulation.<n>Our framework features four powerful alignment components and a user pool of 10 million real individuals.<n>Results demonstrate that SocioVerse can reflect large-scale population dynamics while ensuring diversity, credibility, and representativeness.
  arXiv  Detail & Related papers  (2025-04-14T12:12:52Z)
• MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulations [16.780177153401628]
  We present a novel, open-source social network simulation framework, MOSAIC, where generative language agents predict user behaviors such as liking, sharing, and flagging content.<n>This simulation combines LLM agents with a directed social graph to analyze emergent deception behaviors and gain a better understanding of how users determine the veracity of online social content.
  arXiv  Detail & Related papers  (2025-04-10T15:06:54Z)
• Large Language Model Driven Agents for Simulating Echo Chamber Formation [5.6488384323017]
  The rise of echo chambers on social media platforms has heightened concerns about polarization and the reinforcement of existing beliefs.<n>Traditional approaches for simulating echo chamber formation have often relied on predefined rules and numerical simulations.<n>We present a novel framework that leverages large language models (LLMs) as generative agents to simulate echo chamber dynamics.
  arXiv  Detail & Related papers  (2025-02-25T12:05:11Z)
• AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society [32.849311155921264]
  We propose AgentSociety, a large-scale social simulator that integrates a realistic societal environment.<n>Based on the proposed simulator, we generate social lives for over 10k agents, simulating their 5 million interactions.<n>We focus on four key social issues: polarization, the spread of inflammatory messages, the effects of universal basic income policies, and the impact of external shocks such as hurricanes.
  arXiv  Detail & Related papers  (2025-02-12T15:27:07Z)
• Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents [18.961470450132637]
  This paper emphasizes the importance of spontaneous phenomena, wherein agents deeply engage in contexts and make adaptive decisions without explicit directions. We explored spontaneous cooperation across three competitive scenarios and successfully simulated the gradual emergence of cooperation.
  arXiv  Detail & Related papers  (2024-02-19T18:00:53Z)
• LLM-driven Imitation of Subrational Behavior : Illusion or Reality? [3.2365468114603937]
  Existing work highlights the ability of Large Language Models to address complex reasoning tasks and mimic human communication. We propose to investigate the use of LLMs to generate synthetic human demonstrations, which are then used to learn subrational agent policies. We experimentally evaluate the ability of our framework to model sub-rationality through four simple scenarios.
  arXiv  Detail & Related papers  (2024-02-13T19:46:39Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)
• Training Socially Aligned Language Models on Simulated Social Interactions [99.39979111807388]
  Social alignment in AI systems aims to ensure that these models behave according to established societal values. Current language models (LMs) are trained to rigidly replicate their training corpus in isolation. This work presents a novel training paradigm that permits LMs to learn from simulated social interactions.
  arXiv  Detail & Related papers  (2023-05-26T14:17:36Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.