Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

• URL: http://arxiv.org/abs/2608.29803v1
• Date: Sun, 30 Aug 2026 14:08:06 GMT
• Title: Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments
• Abstract summary: Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations.<n>We conduct a systematic comparison using a naturally occurring online persuasion corpus.<n>Our results show that LLMs achieve only slight agreement with humans.
• Score: 10.717755637740948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed as proxies for human participants in social simulations, yet whether they update their beliefs in response to persuasive arguments, as humans do, remains poorly understood. We conduct a systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view. Our results show that LLMs achieve only slight agreement with humans (Cohen's kappa ranging from 0.079 to 0.178). Content-level analyses show that humans and LLMs agree on the strongest persuasion cues but diverge on finer ones: humans are more swayed by novel content and assertive language, whereas LLMs favor topical similarity and surface-level formatting. At the level of persuasion strategy, LLMs underweight emotional appeals and overweight credibility signals relative to humans, while the type of proposition under debate exerts no measurable effect on the degree of divergence. Furthermore, switching from first-person role-playing to third-person observation shifts all models toward greater resistance to persuasion, with the effect varying across persuasion strategies and textual features. These findings highlight the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and humans process persuasive discourse. Our code is available at https://github.com/tsinghua-fib-lab/LLM-belief-update-cmv.

Related papers

• Belief Cascades Drive Persuasion in LLM Agent Networks [71.17605142402785]
  We study how goal-directed persuaders shift elicited stances in networks of LLM agents grounded in real-world ego-network topologies.<n>We show that direct exposure reliably predicts next-round stance change in competing runs, and peer relays carry smaller but measurable influence.<n>These results argue for evaluating multi-agent persuasion as a trajectory- and exposure-level process.
  arXiv  Detail & Related papers  (2026-08-25T21:01:19Z)
• "I understand your perspective": LLM Persuasion and Sycophancy through the Lens of Communicative Action Theory [6.227886907896225]
  Large Language Models (LLMs) can generate high-quality arguments, yet their ability to engage in nuanced and persuasive communicative actions remains largely unexplored.<n>This work explores the persuasive potential of LLMs through the framework of Jrgen Habermas' Theory of Communicative Action.
  arXiv  Detail & Related papers  (2026-06-06T09:54:31Z)
• Can LLMs Emulate Human Belief Dynamics? [4.918358353535447]
  LLMs fail to capture initial human belief distributions and tend to be overall more conformist than humans.<n>They also take a nuanced approach to emulating human homophilic tendencies within networks.
  arXiv  Detail & Related papers  (2026-05-05T23:21:11Z)
• Who Do LLMs Trust? Human Experts Matter More Than Other LLMs [4.125187280299246]
  Large language models (LLMs) increasingly operate in environments where they encounter social information such as other agents' answers, tool outputs, or human recommendations.<n>This paper investigates whether LLMs exhibit analogous patterns of influence and whether they privilege feedback from humans over feedback from other LLMs.
  arXiv  Detail & Related papers  (2026-02-14T03:03:29Z)
• Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks [52.098988739649705]
  This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater.<n>We develop a no-consensus'' benchmark by curating examples that encompass a variety of a priori ambivalent scenarios.<n>Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters.
  arXiv  Detail & Related papers  (2025-05-28T01:31:54Z)
• Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance [73.19687314438133]
  We study how reliance is affected by contextual features of an interaction. We find that contextual characteristics significantly affect human reliance behavior. Our results show that calibration and language quality alone are insufficient in evaluating the risks of human-LM interactions.
  arXiv  Detail & Related papers  (2024-07-10T18:00:05Z)
• Large Language Models are as persuasive as humans, but how? About the cognitive effort and moral-emotional language of LLM arguments [0.0]
  Large Language Models (LLMs) are already as persuasive as humans. This paper investigates the persuasion strategies of LLMs, comparing them with human-generated arguments.
  arXiv  Detail & Related papers  (2024-04-14T19:01:20Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks [49.60689355674541]
  A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments. We test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with human participants.
  arXiv  Detail & Related papers  (2023-10-30T15:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.