QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

• URL: http://arxiv.org/abs/2605.27068v1
• Date: Tue, 26 May 2026 14:19:08 GMT
• Title: QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents
• Abstract summary: QUACK is an evaluation framework for auditing the grounding of agent language in multimodal social reasoning.<n>It reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it.<n>We find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence.
• Score: 38.13248430205106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.

Related papers

• Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions [66.7908338529737]
  Strategic deception by LLM and VLM agents has emerged as a central AI alignment and safety concern.<n>Existing testbeds are text-only and run on a single fixed agent configuration, missing the non-verbal sensorimotor channels treated as core by deception.<n>We introduce MineAmongUs, a 3D multimodal Among Us sandbox where imposter agents must deceive crewmates through joint verbal and non-verbal action.
  arXiv  Detail & Related papers  (2026-08-31T08:24:29Z)
• CompanionBench: A Theory-Anchored, Real-World-Grounded Benchmark for AI Emotional Companionship [7.643253506288442]
  Existing benchmarks use hand-authored scenarios and prompted simulators.<n>We introduce CompanionBench, an interactive bilingual benchmark.<n>It is the first companion benchmark to ground both its scenarios and a trained user simulator in real-world data.
  arXiv  Detail & Related papers  (2026-08-03T10:44:04Z)
• MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents [7.0922719251235655]
  MetaSpace is a novel framework designed to evaluate the spatial cognition of embodied agents.<n>We show that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents.<n>Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.
  arXiv  Detail & Related papers  (2026-07-26T14:07:20Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions [6.357772907811544]
  We evaluate 22,500 deterministic trajectories across 3 dataset contexts with 3 state-of-the-art (SOTA) models.<n>We formalize the textitSovereignty Gap: models frequently compute the correct derivation internally but suffer Alignment Hallucinations''<n>We prove that multi-agent social load is strictly non-commutative; the "brand" identity of the Lead Anchor'' auditor disproportionately dictates the swarm's integrity.
  arXiv  Detail & Related papers  (2026-05-11T15:13:01Z)
• TriEx: A Game-based Tri-View Framework for Explaining Internal Reasoning in Multi-Agent LLMs [16.622300148265275]
  TriEx is a tri-view explainability framework that instruments sequential decision making with aligned artifacts.<n>We show that TriEx enables scalable analysis of explanation faithfulness, belief dynamics, and evaluator reliability.
  arXiv  Detail & Related papers  (2026-04-21T22:55:57Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• Lying to Win: Assessing LLM Deception through Human-AI Games and Parallel-World Probing [16.419761149171215]
  We introduce a logically grounded framework to elicit and quantify deceptive behavior by embedding Large Language Models.<n>Our method employs a conversational forking mechanism: at the point of object identification, the dialogue state is duplicated into multiple parallel worlds.<n>We evaluate GPT-4o, Gemini-2.5-Flash, and Qwen-3-235B across three incentive levels: neutral, loss-based, and existential.
  arXiv  Detail & Related papers  (2026-03-07T13:21:53Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models [75.85319609088354]
  Sentient Agent as a Judge (SAGE) is an evaluation framework for large language models.<n>SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction.<n>SAGE provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
  arXiv  Detail & Related papers  (2025-05-01T19:06:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.