PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models

• URL: http://arxiv.org/abs/2604.08340v1
• Date: Thu, 09 Apr 2026 15:12:36 GMT
• Title: PokeGym: A Visually-Driven Long-Horizon Benchmark for Vision-Language Models
• Abstract summary: We introduce PokeGym, a visually-driven benchmark instantiated within Pokemon Legends: Z-A, a visually complex 3D open-world Role-Playing Game.<n> PokeGym enforces strict code-level isolation: agents operate solely on raw RGB observations while an independent evaluator verifies success via memory scanning.<n>The benchmark comprises 30 tasks (30-220 steps) spanning navigation, interaction, and mixed scenarios, with three instruction granularities (Visual-Guided, Step-Guided, Goal-Only) to systematically deconstruct visual grounding, semantic reasoning, and autonomous exploration capabilities.
• Score: 33.89873575295867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Vision-Language Models (VLMs) have achieved remarkable progress in static visual understanding, their deployment in complex 3D embodied environments remains severely limited. Existing benchmarks suffer from four critical deficiencies: (1) passive perception tasks circumvent interactive dynamics; (2) simplified 2D environments fail to assess depth perception; (3) privileged state leakage bypasses genuine visual processing; and (4) human evaluation is prohibitively expensive and unscalable. We introduce PokeGym, a visually-driven long-horizon benchmark instantiated within Pokemon Legends: Z-A, a visually complex 3D open-world Role-Playing Game. PokeGym enforces strict code-level isolation: agents operate solely on raw RGB observations while an independent evaluator verifies success via memory scanning, ensuring pure vision-based decision-making and automated, scalable assessment. The benchmark comprises 30 tasks (30-220 steps) spanning navigation, interaction, and mixed scenarios, with three instruction granularities (Visual-Guided, Step-Guided, Goal-Only) to systematically deconstruct visual grounding, semantic reasoning, and autonomous exploration capabilities. Our evaluation reveals a key limitation of current VLMs: physical deadlock recovery, rather than high-level planning, constitutes the primary bottleneck, with deadlocks showing a strong negative correlation with task success. Furthermore, we uncover a metacognitive divergence: weaker models predominantly suffer from Unaware Deadlocks (oblivious to entrapment), whereas advanced models exhibit Aware Deadlocks (recognizing entrapment yet failing to recover). These findings highlight the need to integrate explicit spatial intuition into VLM architectures. The code and benchmark will be available on GitHub.

Related papers

• VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning [133.66021945481876]
  We introduce VBVR-Pro, a testbed that makes native visual reasoning generation controllable.<n> VBVR-Pro turns visual reasoning into a controlled task space of 300 procedurally generated tasks.<n>We identify recurring failure modes of the prevalent VLM-as-a-judge paradigm.
  arXiv  Detail & Related papers  (2026-08-26T17:59:51Z)
• VistaVLA: Geometry- and Semantic-Aware 3D Gaussian-Grounded VLA for Robotic Manipulation [7.753769810587787]
  Vision-Language-Action (VLA) models have emerged as a powerful end-to-end paradigm for robotic manipulation by mapping language instructions and 2D visual inputs directly to actions.<n>These models lack an explicit, scene-level 3D representation, limiting their ability to reason over spatial layouts and geometric constraints.<n>We present VistaVLA, a novel framework that constructs a geometry- and semantics-aware 3D cognitive representation from 3D Gaussian primitives.
  arXiv  Detail & Related papers  (2026-07-14T05:08:50Z)
• 3DVLA: Enhancing Vision-Language-Action Models via 3D Spatial and Instance Understanding [11.30785902722196]
  3DVLA is a plug-and-play framework that injects robust 3D reasoning into pretrained Vision-Language-Action models.<n>We show consistent and significant gains in manipulation performance, validating both the effectiveness and plug-and-play compatibility of our approach.
  arXiv  Detail & Related papers  (2026-05-28T06:07:57Z)
• PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction [57.63073414949329]
  We propose PointACT, a dual-system 3D-aware VLA policy that integrates hierarchical 3D point cloud representations directly into the action decoding process.<n>PointACT employs a multi-scale point-action interaction mechanism with efficient bottleneck window self-attention, enabling evolving action tokens to densely attend to both local geometric detail and global scene structure.
  arXiv  Detail & Related papers  (2026-05-20T17:10:31Z)
• SleepWalk: A Three-Tier Benchmark for Stress-Testing Instruction-Guided Vision-Language Navigation [18.453985392979785]
  SleepWalk is a benchmark for evaluating instruction-grounded trajectory prediction in single-scene 3D worlds.<n>We evaluate three frontier Vision-Language Models on 2,472 curated 3D environments with nine instructions per scene.
  arXiv  Detail & Related papers  (2026-05-11T11:20:14Z)
• 3DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting [12.057873540714098]
  3DGSNav is a novel framework that embeds 3D Gaussian Splatting (3DGS) as persistent memory for vision-language models (VLMs) to enhance spatial reasoning.<n>3DGSNav incrementally constructs a 3DGS representation of the environment, enabling trajectory-guided free-viewpoint rendering of frontier-aware first-person views.<n>During navigation, a real-time object detector filters potential targets, while VLM-driven active viewpoint switching performs target re-verification.
  arXiv  Detail & Related papers  (2026-02-12T16:41:26Z)
• RADAR: Benchmarking Vision-Language-Action Generalization via Real-World Dynamics, Spatial-Physical Intelligence, and Autonomous Evaluation [76.22852262683746]
  We introduce RADAR, a benchmark designed to systematically evaluate VLA generalization under realistic conditions.<n>We apply RADAR to audit multiple state-of-the-art VLA models and uncover severe fragility beneath their apparent competence.
  arXiv  Detail & Related papers  (2026-02-11T16:08:30Z)
• LIBERO-X: Robustness Litmus for Vision-Language-Action Models [32.29541801424534]
  This work systematically rethinks VLA benchmarking from both evaluation and data perspectives.<n>We introduce LIBERO-X, a benchmark featuring a hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities.<n> Experiments with representative VLA models reveal significant performance drops under cumulative perturbations.
  arXiv  Detail & Related papers  (2026-02-06T09:59:12Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• RoboBrain 2.5: Depth in Sight, Time in Mind [63.825215214332104]
  We introduce RoboBrain 2.5, a next-generation embodied AI foundation model that advances general perception, spatial reasoning, and temporal modeling.<n>Specifically, it unlocks Precise 3D Reasoning by shifting from 2D pixel-relative grounding to depth-aware coordinate prediction.<n>Dense Temporal Value Estimation provides dense, step-aware progress prediction and execution state understanding.
  arXiv  Detail & Related papers  (2026-01-20T17:21:54Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal Reasoning [62.09195763860549]
  Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models (LLMs) but struggles with exploration.<n>We introduce $textbfVOGUE (Visual Uncertainty Guided Exploration)$, a novel method that shifts exploration from the output (text) to the input (visual) space.<n>Our work shows that grounding exploration in the inherent uncertainty of visual inputs is an effective strategy for improving multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-01T20:32:08Z)
• OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding [50.72259772580637]
  We introduce OST-Bench, a benchmark designed to evaluate Online Spatio-Temporal understanding from the perspective of an agent.<n>Built on an efficient data collection pipeline, OST-Bench consists of 1.4k scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and ARKitScenes.<n>We find that both complex-based spatial reasoning demands and long-term memory retrieval requirements significantly drop model performance along two separate axes.
  arXiv  Detail & Related papers  (2025-07-10T17:56:07Z)
• VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs [18.349695067647012]
  Visual Language Models excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple tests.<n>We present an evaluation that tests vision-language models' capacity for nonlocal visual reasoning.<n>Our findings show that despite gains in raw visual acuity, current models lack core visual reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-04T23:15:52Z)
• The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering [42.09744951074433]
  We investigate the internal dynamics of hallucination by examining the tokens logits ranking throughout the generation process.<n>We propose VISTA, a training-free inference-time intervention framework that reduces hallucination while promoting genuine information.
  arXiv  Detail & Related papers  (2025-02-05T21:34:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.