S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence
Abstract Overview
S-Agent is a spatial tool-use framework designed for reasoning over continuous multi-view images and videos rather than isolated frames. It treats a vision-language model as a semantic planner that determines necessary evidence, while utilizing a hierarchy of tools to ground objects in 2D, lift observations into 3D geometric evidence, and aggregate this data into higher-level spatial knowledge, such as measurements and relative positions. Furthermore, S-Agent maintains two forms of temporal memory, Scene Memory and Agent Memory, to preserve scene state and reasoning history, enabling it to act both as a training-free inference framework and as a trajectory generator for fine-tuning smaller models.
Novelty
The framework distinguishes itself by formulating video and multi-view spatial reasoning as spatio-temporal evidence accumulation with explicit tool use and persistent memory, rather than relying on stateless, frame-wise prediction. It also innovates by distilling this complex inference-time framework into an S-300K trajectory dataset to effectively train a more compact spatial agent.
Results
In zero-shot evaluations, S-Agent consistently enhances strong VLMs, achieving 46.4% on MMSI-Bench, 60.0% on ViewSpatial-Bench, and 58.8 on ReVSI. It notably outperforms GPT-5.4 by 4.5% on MMSI-Bench and 14.4% on ViewSpatial-Bench. Distilling generated trajectories into S-Agent-8B significantly improves the Qwen3-VL-8B baseline from 31.1% to 41.6% on MMSI-Bench, establishing it as a highly competitive compact spatial agent.
Key Points
- S-Agent separates semantic planning from evidence acquisition using a three-level hierarchy for 2D grounding, 3D geometric lifting, and expert-based spatial aggregation.
- A dual-memory system consisting of Scene Memory and Agent Memory preserves persistent spatial evidence and reasoning history, enabling stateful spatial reasoning across viewpoints and reasoning steps.
- Comprehensive ablations confirm that expert-mediated 3D evidence and both memory modules are crucial for success, while distillation experiments prove generated trajectories successfully train robust, compact 8B agents.
References
- arXiv: https://arxiv.org/abs/2606.20515v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.20515v1
- Hugging Face Papers: https://huggingface.co/papers/2606.20515
- Project: https://Ropedia.github.io/S-Agent