Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym

• URL: http://arxiv.org/abs/2604.09338v1
• Date: Fri, 10 Apr 2026 14:05:50 GMT
• Title: Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym
• Abstract summary: Spatial-Gym tests pathfinding in 2D-grid puzzles as a sequential decision task with optional backtracking.<n>We evaluate eight models in three settings (one-shot, step-by-step, step-by-step with backtracking) against human, random, and A* baselines on 500 episodes.
• Score: 13.424353009036068
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Spatial reasoning is central to navigation and robotics, yet measuring model capabilities on these tasks remains difficult. Existing benchmarks evaluate models in a one-shot setting, requiring full solution generation in a single response, unlike humans, who work in interactive environments step-by-step. We introduce Spatial-Gym, a Gymnasium environment that isolates spatial constraint reasoning by testing pathfinding in 2D-grid puzzles as a sequential decision task with optional backtracking. We evaluate eight models in three settings (one-shot, step-by-step, step-by-step with backtracking) against human, random, and A* baselines on 500 episodes. The best model, GPT-OSS 120B, achieves a solve rate of 16.0%, 82 points below the human baseline (98.0%). Step-by-step format helps weaker models (up to +5.4%) by removing formatting errors, but hurts stronger models (up to 5.6%) by constraining global planning. Backtracking improves episode completion, but increases solve rate only for weaker models; stronger models rarely backtrack and do not benefit from it. Our experiments have three key findings: (1) models fail to scale reasoning effort with difficulty, (2) vision models receiving images of the spatial environment reduce solve rate by 73%, and (3) extended chain-of-thought reasoning retains a 3-5x accuracy advantage over standard inference even in the step-by-step setting. Spatial-Gym enables diagnosis of model limitations and provides a framework for improving spatial reasoning through reinforcement learning.

Related papers

• Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation [89.51408170909548]
  We introduce Explorative Modeling, a new paradigm that factors the training loop, exploring K candidate matches between model generations and data, and training on the best.<n>First, increasing exploration adds a third pretraining axis beyond parameters and data for existing generative models.<n>Second, XMs enable end-to-end reconstructive generative modeling, matching diffusion on control tasks with 16-256x fewer inference steps.
  arXiv  Detail & Related papers  (2026-07-29T18:25:17Z)
• Scaling Self-Play with Self-Guidance [62.13619253976748]
  Self-Guided Self-Play (SGS) is a self-play algorithm that guides the Conjecturer away from degeneracy.<n>SGS surpasses the solve rate of our strongest baseline in fewer than 80 rounds of self-play.
  arXiv  Detail & Related papers  (2026-04-22T05:50:23Z)
• Pioneer Agent: Continual Improvement of Small Language Models in Production [6.452260317191361]
  Small language models are attractive for production deployment due to their low cost, fast inference, and ease of specialization.<n>We present Pioneer Agent, a closed-loop system that automates the process of adapting small language models to a specific task.
  arXiv  Detail & Related papers  (2026-04-10T18:13:09Z)
• Brevity Constraints Reverse Performance Hierarchies in Language Models [0.0]
  On 7.7% of benchmark problems, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters.<n>We identify the mechanism as spontaneous scale-dependent verbosity that introduces errors through overelaboration.<n>Our results establish that maximizing large model performance requires scale-aware prompt engineering.
  arXiv  Detail & Related papers  (2026-03-11T06:47:41Z)
• How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs [49.61011897610774]
  How2Everything is a framework to evaluate and improve goal-conditioned procedure generation.<n>Our framework includes How2Mine, which mines 351K procedures from 980K web pages across 14 topics.<n>How2Score is an evaluation protocol that uses an LLM judge to detect whether a generation contains any critical failure that would prevent achieving the goal.
  arXiv  Detail & Related papers  (2026-02-09T15:47:14Z)
• Chain of Simulation: A Dual-Mode Reasoning Framework for Large Language Models with Dynamic Problem Routing [0.0]
  Chain of Simulation (CoS) is a novel dual-mode reasoning framework that dynamically routes problems to specialized reasoning strategies.<n>CoS employs three distinct reasoning modes: computational flow with self-consistency for mathematical problems, symbolic state tracking with representations for spatial reasoning, and hybrid fact-extraction for multi-hop inference.
  arXiv  Detail & Related papers  (2026-02-02T21:44:01Z)
• Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models [2.3991974633684854]
  Large language models must satisfy hard orthographic constraints during controlled text generation.<n>We evaluate 28 configurations spanning three model families on 58 word puzzles requiring character-level constraint satisfaction.
  arXiv  Detail & Related papers  (2025-11-26T06:12:33Z)
• Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning [49.290631188365786]
  Scaf-GRPO is a training framework that intervenes when a model's independent learning has plateaued.<n>It boosts the pass@1 score of the Qwen2.5-Math-7B model by a relative 44.3% over a vanilla GRPO baseline.<n>This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach.
  arXiv  Detail & Related papers  (2025-10-22T17:41:30Z)
• SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models [73.19077622773075]
  We present a comprehensive methodology for building spatial intelligence progressively.<n>We introduce SpatialLadder-26k, a multimodal dataset containing 26,610 samples spanning object localization, single image, multi-view, and video spatial reasoning tasks.<n>We design a three-stage progressive training framework that establishes spatial perception through object localization, develops spatial understanding through multi-dimensional spatial tasks, and strengthens complex reasoning via reinforcement learning with verifiable rewards.
  arXiv  Detail & Related papers  (2025-10-09T17:50:54Z)
• ReasonGRM: Enhancing Generative Reward Models through Large Reasoning Models [9.30148520355391]
  We present ReasonGRM, a three-stage generative reward modeling framework.<n>In the first stage, Zero-RL is used to generate concise, outcome-directed reasoning paths.<n>In the second stage, $Rstar$, which scores reasoning paths based on their generation likelihood.<n>In the final stage, the model is further refined through reinforcement learning to enhance its preference discrimination capabilities.
  arXiv  Detail & Related papers  (2025-06-20T03:10:52Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction [89.56181323849512]
  SuperCorrect is a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model.<n>In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts.<n>In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model.
  arXiv  Detail & Related papers  (2024-10-11T17:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.