Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions

• URL: http://arxiv.org/abs/2608.30428v1
• Date: Mon, 31 Aug 2026 08:24:29 GMT
• Title: Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions
• Abstract summary: Strategic deception by LLM and VLM agents has emerged as a central AI alignment and safety concern.<n>Existing testbeds are text-only and run on a single fixed agent configuration, missing the non-verbal sensorimotor channels treated as core by deception.<n>We introduce MineAmongUs, a 3D multimodal Among Us sandbox where imposter agents must deceive crewmates through joint verbal and non-verbal action.
• Score: 66.7908338529737
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Strategic deception by LLM and VLM agents has emerged as a central AI alignment and safety concern. Social-deduction games (where each player holds a hidden role and communicates with others to deduce identities) serve as the canonical testbed, particularly in multi-agent settings. Existing testbeds, however, are text-only and run on a single fixed agent configuration, missing the non-verbal sensorimotor channels treated as core by deception taxonomies and leaving it ambiguous whether an observed behavior reflects the underlying model or the surrounding harness. We introduce MineAmongUs, a 3D multimodal Among Us sandbox where imposter agents must deceive crewmates through joint verbal and non-verbal action. We also propose ARIA, a configurable VLM-agent harness that exposes five cognitive-component ablation axes; and an atom- and arc-level annotation scheme grounded in deception taxonomies and operationalized at scale by an LLM-as-a-Judge reaching near-human atom-labeling agreement. Empirical results show that VLM agents pursue imposter wins through joint verbal and non-verbal deception, with non-verbal channels emerging as the more decisive winning contributors across both harness ablation and cross-VLM evaluation. Taken together, our work opens a new path for embodied VLM-agent alignment research.

Related papers

• Belief Cascades Drive Persuasion in LLM Agent Networks [71.17605142402785]
  We study how goal-directed persuaders shift elicited stances in networks of LLM agents grounded in real-world ego-network topologies.<n>We show that direct exposure reliably predicts next-round stance change in competing runs, and peer relays carry smaller but measurable influence.<n>These results argue for evaluating multi-agent persuasion as a trajectory- and exposure-level process.
  arXiv  Detail & Related papers  (2026-08-25T21:01:19Z)
• When Collaboration Becomes a Trigger: Collective Evidence-Threshold Backdoors in Multi-Agent Systems [57.450132281381066]
  Backdoor behavior can be activated when peer evidence reaches a hidden threshold, rather than being determined by any single message.<n>We introduce a collective evidence-threshold backdoor paradigm for MAS and Boundary-Conditioned Backdoor Injection (BCBI)<n>We propose LAtent Transition Test-time Evaluation (LATTE), a clean-only latent-transition defense that learns benign communication dynamics and quarantines anomalous agent updates.
  arXiv  Detail & Related papers  (2026-08-02T08:38:44Z)
• Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems [55.51100373104311]
  We present Colosseum, a framework for auditing LLM agents' collusive behavior in multi-agent settings.<n>Colosseum tests each LLM for collusion under different objectives, persuasion tactics, and network topologies.<n>We discover collusion on paper'' when agents plan to collude in text but would often pick non-collusive actions.
  arXiv  Detail & Related papers  (2026-02-16T21:27:38Z)
• When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agents [45.43445469098021]
  This paper presents a game-theoretic study of covert communication in multi-agent systems.<n>We analyse interactions across four canonical game-theoretic settings under different communication regimes.<n>Considering heterogeneous agent personalities and both one-shot and repeated games, we characterise when covert signals emerge and how they shape coordination and strategic outcomes.
  arXiv  Detail & Related papers  (2026-01-07T12:07:48Z)
• Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry [17.472005826931127]
  This paper studies Large Language Model (LLM) agents in task collaboration.<n>We extend Einstein Puzzles, a symbolic puzzle, to a table-top game.<n> Empirical results highlight the critical importance of aligned communication.
  arXiv  Detail & Related papers  (2025-10-29T15:03:53Z)
• Opponent Shaping in LLM Agents [9.180524457769751]
  We present the first investigation of opponent shaping (OS) with Large Language Models (LLMs)<n>Using ShapeLLM, we examine whether LLM agents can influence co-players' learning dynamics across diverse game-theoretic environments.<n>Our findings show that LLM agents can both shape and be shaped through interaction, establishing opponent shaping as a key dimension of multi-agent LLM research.
  arXiv  Detail & Related papers  (2025-10-09T14:13:24Z)
• The Traitors: Deception and Trust in Multi-Agent Language Model Simulations [0.0]
  We introduce The Traitors, a multi-agent simulation framework inspired by social deduction games.<n>We develop a suite of evaluation metrics capturing deception success, trust dynamics, and collective inference quality.<n>Our initial experiments across DeepSeek-V3, GPT-4o-mini, and GPT-4o (10 runs per model) reveal a notable asymmetry.
  arXiv  Detail & Related papers  (2025-05-19T10:01:35Z)
• Among Us: A Sandbox for Measuring and Detecting Agentic Deception [1.1893676124374688]
  We introduce $textitAmong Us$, a social deception game where language-based agents exhibit long-term, open-ended deception.<n>We find that models trained with RL are comparatively much better at producing deception than detecting it.<n>We also find two SAE features that work well at deception detection but are unable to steer the model to lie less.
  arXiv  Detail & Related papers  (2025-04-05T06:09:32Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay [55.12945794835791]
  Using Avalon as a testbed, we employ system prompts to guide LLM agents in gameplay. We propose a novel framework, tailored for Avalon, features a multi-agent system facilitating efficient communication and interaction. Results affirm the framework's effectiveness in creating adaptive agents and suggest LLM-based agents' potential in navigating dynamic social interactions.
  arXiv  Detail & Related papers  (2023-10-23T14:35:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.