GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents

• URL: http://arxiv.org/abs/2603.24329v1
• Date: Wed, 25 Mar 2026 14:10:45 GMT
• Title: GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents
• Abstract summary: We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding.<n>We densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World.<n>From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy.
• Score: 4.920953895710103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective, capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World, a natural decomposition for multi-agent environments. From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.

Related papers

• Agentic Collaborative Cognition for Zero-Shot 3D Understanding [57.854375071638856]
  We propose a collaborative multi-agent framework that assigns a Planning Agent to handle high-level viewpoint planning.<n>Perception Agent to explicitly summarize the 3D scene into a structured holistic cognitive map.<n>Our method achieves state-of-the-art performance on 6 benchmarks, with improvements of 11.1% Acc@0.5 on ScanRefer, 14.6 BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D.
  arXiv  Detail & Related papers  (2026-06-23T14:44:04Z)
• Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players [87.87842088508553]
  We present our generative multi-agent world model for interactive simulation.<n>Our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines.
  arXiv  Detail & Related papers  (2026-05-27T17:59:31Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents [54.48066948369172]
  As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future.<n>Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video.<n>We first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents.
  arXiv  Detail & Related papers  (2026-03-10T15:48:35Z)
• Proact-VL: A Proactive VideoLLM for Real-Time AI Companions [52.23988809605433]
  We instantiate AI companions through two gaming scenarios, commentator and guide, selected for automatic evaluation.<n>We present Proact-VL, a framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction.
  arXiv  Detail & Related papers  (2026-03-03T19:02:46Z)
• Dyna-Mind: Learning to Simulate from Experience for Better AI Agents [62.21219817256246]
  We argue that current AI agents need ''vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting.<n>We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning.
  arXiv  Detail & Related papers  (2025-10-10T17:30:18Z)
• AVA: Attentive VLM Agent for Mastering StarCraft II [56.07921367623274]
  We introduce Attentive VLM Agent (AVA), a multimodal StarCraft II agent that aligns artificial agent perception with the human gameplay experience.<n>Our agent addresses this limitation by incorporating RGB visual inputs and natural language observations that more closely simulate human cognitive processes during gameplay.
  arXiv  Detail & Related papers  (2025-03-07T12:54:25Z)
• A Survey on Large Language Model-Based Game Agents [35.34074811680046]
  Game agents offer a valuable testbed for exploring capabilities relevant to Artificial General Intelligence.<n>Recently, the emergence of Large Language Models (LLMs) provides new opportunities to endow these agents with generalizable reasoning.<n>This survey offers an up-to-date review of LLM-based game agents through a unified reference architecture.
  arXiv  Detail & Related papers  (2024-04-02T15:34:18Z)
• Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding [61.57847727651068]
  Temporal sentence grounding aims to localize a target segment in an untrimmed video semantically according to a given sentence query. Most previous works focus on learning frame-level features of each whole frame in the entire video, and directly match them with the textual information. We propose a novel Motion- and Appearance-guided 3D Semantic Reasoning Network (MA3SRN), which incorporates optical-flow-guided motion-aware, detection-based appearance-aware, and 3D-aware object-level features.
  arXiv  Detail & Related papers  (2022-03-06T13:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.