OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

• URL: http://arxiv.org/abs/2606.09826v1
• Date: Mon, 08 Jun 2026 17:59:43 GMT
• Title: OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
• Abstract summary: Vision-language model (VLM) agents are increasingly deployed in interactive game environments.<n>Game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes.<n>We address these gaps with OmniGameArena, a real-time benchmark of twelve newly built Unreal Engine 5 games spanning Solo (7), variants (3), and Coop (2) with unified action interfaces, and the Improvement Dynamics Curve (IDC), an agentic-reflection harness.
• Score: 39.555682739423794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes (commercial VLMs, open-weight VLMs, and specialized game policies) on the same footing. We address these gaps with OmniGameArena, a real-time benchmark of twelve newly built Unreal Engine 5 games spanning Solo (7), PvP (3), and Coop (2) with unified action interfaces, and the Improvement Dynamics Curve (IDC), an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds. Beyond cold-start leaderboard scores, IDC exposes two additional observables for each (agent, game) pair: how the score evolves across reflection rounds, and how the learned skill behaves on held-out task variants. We report these observables for twelve VLM agents on the cold-start leaderboard and four top agents under IDC.

Related papers

• Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players [87.87842088508553]
  We present our generative multi-agent world model for interactive simulation.<n>Our model improves video fidelity, action controllability, and inter-agent consistency over slot-based and dense-attention baselines.
  arXiv  Detail & Related papers  (2026-05-27T17:59:31Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents [4.920953895710103]
  We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding.<n>We densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World.<n>From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy.
  arXiv  Detail & Related papers  (2026-03-25T14:10:45Z)
• MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games [79.72300527041534]
  Multi-agent game evaluations often exhibit substantial run-to-run variance.<n>Small deviations compound across turns and are amplified by multi-agent coupling.<n>We address both instability and underperformance with MEMO (Memory-augmented MOdel context optimization), a self-play framework.
  arXiv  Detail & Related papers  (2026-03-09T23:36:32Z)
• LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition [104.81487689011341]
  We introduce LM Fight Arena, a novel framework that evaluates large multimodal models in Mortal Kombat II.<n>Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-10T02:19:21Z)
• Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games [29.194229891848853]
  Orak is a benchmark designed to train and evaluate Large Language Model (LLM) agents across diverse real-world video games.<n>To support consistent evaluation of LLMs, we introduce a plug-and-play interface based on Model Context Protocol (MCP)<n>Orak offers a comprehensive evaluation framework, encompassing general game score leaderboards, LLM battle arenas, and in-depth analyses of visual input state, agentic strategies, and fine-tuning effects.
  arXiv  Detail & Related papers  (2025-06-04T06:40:33Z)
• Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents [33.899782380901314]
  VLM-based mobile agents are increasingly popular due to their capabilities to interact with smartphone GUIs and XML-structured texts.<n>Existing online benchmarks struggle with obtaining stable reward signals due to dynamic environmental changes.<n>Mobile-Bench-v2 includes a common task split, with offline multi-path evaluation to assess the agent's ability to obtain step rewards.
  arXiv  Detail & Related papers  (2025-05-17T07:58:34Z)
• Cultivating Game Sense for Yourself: Making VLMs Gaming Experts [23.370716496046217]
  We propose a paradigm shift in gameplay agent design.<n>Instead of directly controlling gameplay, VLM develops specialized execution modules tailored for tasks like shooting and combat.<n>These modules handle real-time game interactions, elevating VLM to a high-level developer.
  arXiv  Detail & Related papers  (2025-03-27T08:40:47Z)
• AVA: Attentive VLM Agent for Mastering StarCraft II [56.07921367623274]
  We introduce Attentive VLM Agent (AVA), a multimodal StarCraft II agent that aligns artificial agent perception with the human gameplay experience.<n>Our agent addresses this limitation by incorporating RGB visual inputs and natural language observations that more closely simulate human cognitive processes during gameplay.
  arXiv  Detail & Related papers  (2025-03-07T12:54:25Z)
• How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments [83.78240828340681]
  GAMA($gamma$)-Bench is a new framework for evaluating Large Language Models' Gaming Ability in Multi-Agent environments.<n>$gamma$-Bench includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to assess LLMs' performance.<n>Our results indicate GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought.
  arXiv  Detail & Related papers  (2024-03-18T14:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.