Evaluating Large Language Models in a Complex Hidden Role Game

• URL: http://arxiv.org/abs/2605.22826v1
• Date: Thu, 09 Apr 2026 14:02:14 GMT
• Title: Evaluating Large Language Models in a Complex Hidden Role Game
• Abstract summary: Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments.<n>This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.

Related papers

• AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes [59.134503227559634]
  Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding.<n>Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language.<n>We propose Affect Omni, a framework for verifiable affective reasoning.
  arXiv  Detail & Related papers  (2026-08-24T11:25:04Z)
• Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game [23.840355539083372]
  Large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems.<n>We evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry.<n>We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency.
  arXiv  Detail & Related papers  (2026-07-30T12:54:17Z)
• Intentional Deception as Controllable Capability in LLM Agents [0.0]
  We present a systematic study of intentional deception as an engineered capability in multi-agent systems.<n>We investigate a two-stage system that infers target agent characteristics and generates deceptive responses steering targets toward actions counter to their beliefs and motivations.<n>We find that deceptive intervention produces differential effects concentrated in specific behavioral profiles rather than distributed uniformly.
  arXiv  Detail & Related papers  (2026-03-08T23:48:49Z)
• Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People [81.63702981397408]
  Given limited resources, to what extent do agents based on language models (LMs) act rationally?<n>We develop methods to benchmark and enhance agentic information-seeking, drawing on insights from human behavior.<n>For Spotter agents, our approach boosts accuracy by up to 14.7% absolute over LM-only baselines; for Captain agents, it raises expected information gain (EIG) by up to 0.227 bits (94.2% of the achievable noise ceiling)
  arXiv  Detail & Related papers  (2025-10-23T17:57:28Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies [54.08697738311866]
  Social deduction games like Werewolf combine language, reasoning, and strategy.<n>We curate a high-quality, human-verified multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants.<n>We propose a novel strategy-alignment evaluation that leverages the winning faction's strategies as ground truth in two stages.
  arXiv  Detail & Related papers  (2025-10-13T13:33:30Z)
• Bayesian Social Deduction with Graph-Informed Language Models [3.7540464038118633]
  Social reasoning remains a challenging task for large language models.<n>We introduce a hybrid reasoning framework that externalizes belief inference to a structured probabilistic model.<n>Our approach achieves competitive performance with much larger models in Agent-Agent play.
  arXiv  Detail & Related papers  (2025-06-21T18:45:28Z)
• SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning [99.645427839457]
  Self-Play Critic (SPC) is a novel approach where a critic model evolves its ability to assess reasoning steps through adversarial self-play games.<n>SPC involves fine-tuning two copies of a base model to play two roles, namely a "sneaky generator" and a "critic"
  arXiv  Detail & Related papers  (2025-04-27T08:45:06Z)
• Effective faking of verbal deception detection with target-aligned adversarial attacks [0.3441021278275805]
  Automated adversarial attacks that rewrite deceptive statements to appear truthful pose a serious threat.<n>We used a dataset of 243 truthful and 262 fabricated autobiographical stories in a deception detection task for humans and machine learning models.
  arXiv  Detail & Related papers  (2025-01-10T13:42:40Z)
• Better Zero-Shot Reasoning with Role-Play Prompting [10.90357246745529]
  Role-play prompting consistently surpasses the standard zero-shot approach across most datasets. This highlights its potential to augment the reasoning capabilities of large language models.
  arXiv  Detail & Related papers  (2023-08-15T11:08:30Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.