Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games

• URL: http://arxiv.org/abs/2601.13709v1
• Date: Tue, 20 Jan 2026 08:07:21 GMT
• Title: Hidden in Plain Text: Measuring LLM Deception Quality Against Human Baselines Using Social Deduction Games
• Abstract summary: Large Language Model (LLM) agents are increasingly used in many applications, raising concerns about their safety.<n>We study deception in the Social Deduction Game (SDG) Mafia, where success is dependent on deceiving others through conversation.<n>We create a Mafia Detector using GPT-4-Turbo to analyze game transcripts without player role information to predict the mafia players.<n>Results show that the Mafia Detector's mafia prediction accuracy is lower on LLM games than on human games.
• Score: 4.501547677532766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Model (LLM) agents are increasingly used in many applications, raising concerns about their safety. While previous work has shown that LLMs can deceive in controlled tasks, less is known about their ability to deceive using natural language in social contexts. In this paper, we study deception in the Social Deduction Game (SDG) Mafia, where success is dependent on deceiving others through conversation. Unlike previous SDG studies, we use an asynchronous multi-agent framework which better simulates realistic social contexts. We simulate 35 Mafia games with GPT-4o LLM agents. We then create a Mafia Detector using GPT-4-Turbo to analyze game transcripts without player role information to predict the mafia players. We use prediction accuracy as a surrogate marker for deception quality. We compare this prediction accuracy to that of 28 human games and a random baseline. Results show that the Mafia Detector's mafia prediction accuracy is lower on LLM games than on human games. The result is consistent regardless of the game days and the number of mafias detected. This indicates that LLMs blend in better and thus deceive more effectively. We also release a dataset of LLM Mafia transcripts to support future research. Our findings underscore both the sophistication and risks of LLM deception in social contexts.

Related papers

• Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game [23.840355539083372]
  Large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems.<n>We evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry.<n>We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency.
  arXiv  Detail & Related papers  (2026-07-30T12:54:17Z)
• Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models [13.754658024896612]
  We study the abilities of Large Language Models to persuade and be rationally vigilant towards other LLM agents.<n>We find that puzzle-solving performance, persuasive capability, and vigilance are dissociable capacities in LLMs.<n>Our work presents the first investigation of the relationship between persuasion, vigilance, and task performance in LLMs.
  arXiv  Detail & Related papers  (2026-02-24T04:09:21Z)
• Do You Get the Hint? Benchmarking LLMs on the Board Game Concept [1.671764884922859]
  Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses.<n>In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning in a representation that is much closer to natural language data.<n>Our results show that this game, easily solved by humans (with a success rate of over 90%), is still very challenging for state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-10-15T08:17:25Z)
• Deceive, Detect, and Disclose: Large Language Models Play Mini-Mafia [0.0]
  Mini-Mafia is a social deduction game where informed mafia compete against uninformed townsfolk.<n>Its asymmetry of information and reliance on theory-of-mind reasoning mirror real-world multi-agent scenarios.<n>We introduce Mini-Mafia: a simplified four-player variant with one mafioso, one detective, and two villagers.
  arXiv  Detail & Related papers  (2025-09-27T00:40:19Z)
• Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation [66.84286617519258]
  Large language models are transforming social science research by enabling the automation of labor-intensive tasks like data annotation and text analysis.<n>Such variation can introduce systematic biases and random errors, which propagate to downstream analyses and cause Type I (false positive), Type II (false negative), Type S (wrong sign), or Type M (exaggerated effect) errors.<n>We find that intentional LLM hacking is strikingly simple. By replicating 37 data annotation tasks from 21 published social science studies, we show that, with just a handful of prompt paraphrases, virtually anything can be presented as statistically significant.
  arXiv  Detail & Related papers  (2025-09-10T17:58:53Z)
• Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts [79.1081247754018]
  Large Language Models (LLMs) are widely deployed in reasoning, planning, and decision-making tasks.<n>We propose a framework based on Contact Searching Questions(CSQ) to quantify the likelihood of deception.
  arXiv  Detail & Related papers  (2025-08-08T14:46:35Z)
• lmgame-Bench: How Good are LLMs at Playing Games? [60.01834131847881]
  We study the major challenges in using popular video games to evaluate modern large language model (LLM) agents.<n>We introduce lmgame-Bench to turn games into reliable evaluations.
  arXiv  Detail & Related papers  (2025-05-21T06:02:55Z)
• Can Large Language Models Capture Video Game Engagement? [1.3873323883842132]
  We evaluate comprehensively the capacity of popular Large Language Models to annotate and successfully predict continuous affect annotations of videos.<n>We run over 2,400 experiments to investigate the impact of LLM architecture, model size, input modality, prompting strategy, and ground truth processing method on engagement prediction.
  arXiv  Detail & Related papers  (2025-02-05T17:14:47Z)
• GAMEBoT: Transparent Assessment of LLM Reasoning in Games [54.49589494014147]
  GAMEBoT is a gaming arena designed for rigorous assessment of Large Language Models.<n>We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.<n>Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.
  arXiv  Detail & Related papers  (2024-12-18T08:32:53Z)
• LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents [10.632179121247466]
  We propose a general game-testing framework using LLM agents and test it on two widely played strategy games: Wordle and Slay the Spire. Our results reveal an interesting finding: although LLMs may not perform as well as the average human player, their performance, when guided by simple, generic prompting techniques, shows a statistically significant and strong correlation with difficulty indicated by human players. This suggests that LLMs could serve as effective agents for measuring game difficulty during the development process.
  arXiv  Detail & Related papers  (2024-10-01T18:40:43Z)
• GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations [87.99872683336395]
  Large Language Models (LLMs) are integrated into critical real-world applications. This paper evaluates LLMs' reasoning abilities in competitive environments. We first propose GTBench, a language-driven environment composing 10 widely recognized tasks.
  arXiv  Detail & Related papers  (2024-02-19T18:23:36Z)
• LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay [55.12945794835791]
  Using Avalon as a testbed, we employ system prompts to guide LLM agents in gameplay. We propose a novel framework, tailored for Avalon, features a multi-agent system facilitating efficient communication and interaction. Results affirm the framework's effectiveness in creating adaptive agents and suggest LLM-based agents' potential in navigating dynamic social interactions.
  arXiv  Detail & Related papers  (2023-10-23T14:35:26Z)
• Playing repeated games with Large Language Models [23.57695486747895]
  LLMs are increasingly used in applications where they interact with humans and other agents.<n>We let different LLMs play finitely repeated $2times2$ games with each other, with human-like strategies, and actual human players.<n>Our results show that LLMs perform particularly well at self-interested games like the iterated Prisoner's Dilemma family.<n>However, they behave sub-optimally in games that require coordination, like the Battle of the Sexes.
  arXiv  Detail & Related papers  (2023-05-26T12:17:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.