Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

• URL: http://arxiv.org/abs/2605.22283v1
• Date: Thu, 21 May 2026 10:32:53 GMT
• Title: Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action
• Abstract summary: We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models.<n>The framework consists of three components: Spatial Memory Construction, Dynamic Memory Refinement, and Contextual Memory Retrieval.<n>We evaluate SOMA on five challenging real-world out-of-vision manipulation tasks, including multi-step and dual-arm scenarios.
• Score: 33.511005694762126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce SOMA, the Spatial Memory framework for Out-of-Vision Manipulation in Vision-Language-Action (VLA) models. Most existing VLAs implicitly assume that task-relevant objects are always visible, leading to brittle and reactive behaviors when targets fall outside the camera's field of view. SOMA addresses this limitation by equipping VLAs with a persistent spatial memory constructed from multi-view observations acquired via a movable head camera, enabling reasoning beyond the current visual frustum. The framework consists of three components: Spatial Memory Construction, which aggregates angular-wise observations into a unified spatial-semantic representation through scanning; Dynamic Memory Refinement, which maintains global consistency over time; and Contextual Memory Retrieval, which activates instruction-relevant spatial cues during manipulation. We evaluate SOMA on five challenging real-world out-of-vision manipulation tasks, including multi-step and dual-arm scenarios where target objects are initially invisible. Experimental results show that SOMA not only improves task success rates, but also induces qualitatively different manipulation behaviors, with faster target localization, reduced viewpoint search, and near one-shot grasping under partial observability. Additional experiments on RoboCasa GR1 and SimplerEnv further validate the effectiveness of SOMA's memory design under conventional fully observable settings. Code will be released soon.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• MemoryWAM: Efficient World Action Modeling with Persistent Memory [80.90899269062128]
  We introduce MemoryWAM, a world action model with efficient persistent memory.<n>A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context.<n>MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines in both simulation and the real world.
  arXiv  Detail & Related papers  (2026-06-18T17:59:51Z)
• EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies [68.812675280427]
  EventVLA is an end-to-end framework founded on the concept of sparse visual evidence memory.<n>KEM directly predicts future probabilities from the VLA's latent embeddings to autonomously capture and store sparse, task-critical visual events.<n>We propose RoboTwin-MeM, a diagnostic benchmark designed to evaluate non-Markovian manipulation tasks with interactive visual evidence.
  arXiv  Detail & Related papers  (2026-06-18T11:11:37Z)
• GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoning [9.203056739430865]
  GSMem is a zero-shot embodied exploration and reasoning framework built upon 3D Gaussian Splatting (3DGS)<n>By explicitly parameterizing continuous geometry and dense appearance, 3DGS serves as a persistent spatial memory that endows the agent with textitSpatial Recollection<n>We introduce a hybrid exploration strategy that combines VLM-driven semantic scoring with a 3DGS-based coverage objective, balancing task-aware exploration with geometric coverage.
  arXiv  Detail & Related papers  (2026-03-19T16:55:54Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective [16.541717037293278]
  We introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability.<n>It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame.<n>We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability.
  arXiv  Detail & Related papers  (2025-11-14T16:56:01Z)
• VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models [78.88575188716378]
  VisMem is a framework that equips Vision-Language Models with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation.<n>Our experiments reveal that VisMem delivers a significant average performance boost of 11.8% relative to the vanilla model.
  arXiv  Detail & Related papers  (2025-11-14T06:51:34Z)
• Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Models [13.17978215666921]
  Large Vision-Language Models (LVLMs) can accurately locate key objects in images, yet their attention to these objects tends to be very brief.<n>Motivated by the hypothesis that sustained focus on key objects can improve LVLMs' visual capabilities, we propose Cross-Layer Vision Smoothing (CLVS)<n> CLVS achieves state-of-the-art performance on a variety of visual understanding tasks.
  arXiv  Detail & Related papers  (2025-09-16T09:54:01Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.