Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game

• URL: http://arxiv.org/abs/2607.28146v1
• Date: Thu, 30 Jul 2026 12:54:17 GMT
• Title: Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game
• Abstract summary: Large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems.<n>We evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry.<n>We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency.
• Score: 23.840355539083372
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: As large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems, understanding their deceptive capabilities is fundamental to safety. Controlled social deduction games provide a reproducible proxy for isolating and evaluating these complex adversarial behaviors. We present the open-source benchmark framework ParliamentBench based on the game Secret Hitler to evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry. We evaluate 16 LLMs across 1,600 simulated matches playing each other, playing against humans, and compare them against a large set of online games. We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency. Our experiments reveal that frontier models achieve strong performance across cooperative and deceptive roles, with a strong top-four cluster (GPT-5.4, Kimi K2.5, Grok 4.1 Fast, and DeepSeek 3.1 Terminus), whereas the weakest models fall short of random (33%) and simple algorithmic (45%) baselines. Most LLMs struggle to maintain a consistent deceptive persona throughout an entire game, with deception retention dropping below 50%.

Related papers

• Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments [33.6358292917346]
  We introduce Social Gym, an environment of 21 multi-agent social games with a cross-game leaderboard.<n>We then propose SPaRTan, a training-free self-improvement loop.<n>Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance.
  arXiv  Detail & Related papers  (2026-08-10T05:12:56Z)
• Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games [50.880420636090896]
  We build an auditable framework that maintains an external belief state over hidden roles in a 9-player Werewolf environment.<n>We report the effect as an association and treat its mechanism as unresolved.
  arXiv  Detail & Related papers  (2026-07-12T16:03:30Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents [38.13248430205106]
  QUACK is an evaluation framework for auditing the grounding of agent language in multimodal social reasoning.<n>It reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it.<n>We find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence.
  arXiv  Detail & Related papers  (2026-05-26T14:19:08Z)
• LLM Wardens: Mitigating Adversarial Persuasion with Third-Party Conversational Oversight [6.508925629329849]
  We find that an adversarial LLM with a hidden goal succeeds in steering users' decisions 65.4% of the time.<n>We then introduce a secondary LLM that monitors the human-AI interaction trace in real time and issues non-binding, private advisories to the user.<n>To probe the mechanism behind these results, we release COAX-Bench, a simulation benchmark spanning 14 decision-making scenarios.
  arXiv  Detail & Related papers  (2026-05-08T16:23:47Z)
• Evaluating Large Language Models in a Complex Hidden Role Game [0.0]
  Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments.<n>This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler.
  arXiv  Detail & Related papers  (2026-04-09T14:02:14Z)
• LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo [1.5718921092089344]
  LudoBench is a benchmark for evaluating strategic reasoning in Ludo, a multi-agent board game.<n>LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories.<n>All models agree with the game-theory baseline only 40-46% of the time.
  arXiv  Detail & Related papers  (2026-04-07T10:34:13Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies [54.08697738311866]
  Social deduction games like Werewolf combine language, reasoning, and strategy.<n>We curate a high-quality, human-verified multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants.<n>We propose a novel strategy-alignment evaluation that leverages the winning faction's strategies as ground truth in two stages.
  arXiv  Detail & Related papers  (2025-10-13T13:33:30Z)
• lmgame-Bench: How Good are LLMs at Playing Games? [60.01834131847881]
  We study the major challenges in using popular video games to evaluate modern large language model (LLM) agents.<n>We introduce lmgame-Bench to turn games into reliable evaluations.
  arXiv  Detail & Related papers  (2025-05-21T06:02:55Z)
• How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments [83.78240828340681]
  GAMA($gamma$)-Bench is a new framework for evaluating Large Language Models' Gaming Ability in Multi-Agent environments.<n>$gamma$-Bench includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to assess LLMs' performance.<n>Our results indicate GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought.
  arXiv  Detail & Related papers  (2024-03-18T14:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.