Dynamic Resilient Spatio-Semantic Memory with Hybrid Localization for Mobile Manipulation

• URL: http://arxiv.org/abs/2606.00576v1
• Date: Sat, 30 May 2026 06:58:03 GMT
• Title: Dynamic Resilient Spatio-Semantic Memory with Hybrid Localization for Mobile Manipulation
• Abstract summary: This paper presents DREAM, a real-robot mobile manipulation framework that integrates memory, localization, navigation, and manipulation in previously unseen indoor environments without a pre-built map.<n>For target localization and reacquisition, DREAM combines language-conditioned 3D retrieval, open-vocabulary image detection, and multimodal large language model based semantic verification.<n>Real-robot experiments in four dynamic indoor laboratory scenes show that DREAM improves long-horizon task success rates from 40%-60% with DynaMem to 55%-70%, while maintaining a memory footprint of 0.37-0.63 GB and an online memory-update time of 0.43-0.53
• Score: 13.692484702750313
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable mobile manipulation in dynamic indoor environments requires a scene representation that remains geometrically consistent, semantically queryable, and computationally bounded as the environment changes. Existing systems often rely on pre-built maps, static-scene assumptions, or highly accurate camera poses, which can lead to stale or misaligned scene information when target objects are relocated or pose estimates are corrected. This paper presents DREAM, a real-robot mobile manipulation framework that integrates perception, memory, localization, navigation, and manipulation in previously unseen indoor environments without a pre-built map. DREAM constructs an online spatio-semantic voxel memory from RGB-D observations registered by a LiDAR-inertial-visual SLAM backend. It further introduces pose-graph-aware Redundancy-Aware Memory Pruning (RMP) to update historical observations after pose corrections while keeping long-horizon observation history bounded. For target localization and reacquisition, DREAM combines language-conditioned 3D retrieval, open-vocabulary image detection, and multimodal large language model based semantic verification. Real-robot experiments in four dynamic indoor laboratory scenes show that DREAM improves long-horizon task success rates from 40%-60% with DynaMem to 55%-70%, while maintaining a memory footprint of 0.37-0.63 GB and an online memory-update time of 0.43-0.53 s across scenes.

Related papers

• HitMem: Hierarchical Temporal 3D Memory with Multi-Modal Context-Aware Retrieval for Dynamic Environments [5.55387168911911]
  HitMem is a hierarchical temporal 3D memory framework with a multi-modal context-aware retrieval mechanism.<n>We show that HitMem significantly improves object relocation accuracy, reduces exploration costs, and enhances task execution performance in dynamic environments.
  arXiv  Detail & Related papers  (2026-09-01T09:09:21Z)
• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation [55.42006264038458]
  Action-conditioned world models have emerged as a promising paradigm for robot learning.<n>Mem-World is a memory-augmented multi-view action-conditioned world model.<n>We present W-VMem, a 4D wrist-view-centered surfel-indexed memory that anchors historical observations to temporally evolving surface elements.
  arXiv  Detail & Related papers  (2026-06-17T11:42:00Z)
• EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigation [81.54723508469617]
  EvoMemNav is an efficient, self-evolving, fine-grained memory framework for zero-shot embodied navigation.<n>VSMGraph organizes views with semantic cues and topological relations into a room-view-object hierarchy.<n>Experiments on GOAT-Bench and HM3D across object, text-description, and image-goal modalities show consistent gains in SR/SPL.
  arXiv  Detail & Related papers  (2026-06-02T11:27:44Z)
• Reasmory: 3D Reconstruction as Explicit Memory for VLMs Spatial Reasoning [43.950190960277865]
  Vision-Language Models (VLMs) exhibit emerging spatial reasoning capabilities, yet they remain unreliable on tasks requiring precise spatial understanding.<n>We propose textbfReasmory, a framework that formulates spatial reasoning as structured program execution over reconstructed spatial memory.<n>Reasmory constructs explicit 3D memory, augments it with semantically grounded 3D object instances, and introduces a lightweight Domain-Specific Language.
  arXiv  Detail & Related papers  (2026-05-31T02:36:57Z)
• Learning to Evolve: Multi-modal Interactive Fields for Robust Humanoid Navigation in Dynamic Environments [10.149525023566712]
  Multi-modal Interactive Field (MIF) is a humanoid-oriented system that integrates confidence-aware semantic 3D Gaussian Splatting, discrepancy-triggered spatial memory updates, and task-driven geometric reconstruction within a closed-loop perception-adaptation pipeline.<n>On a Unitree-G1 humanoid in a real dynamic office, MIF improves relocation success in non-static environments from 12% to 94% compared with static scene-graph memory, while reducing semantic memory footprint by 91.4% through feature distillation for practical online operation.
  arXiv  Detail & Related papers  (2026-05-21T03:11:43Z)
• Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction [50.5449251266956]
  This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
  arXiv  Detail & Related papers  (2026-04-09T17:59:50Z)
• CogniMap3D: Cognitive 3D Mapping and Rapid Retrieval [13.47989214839101]
  We present CogniMap3D, a bioinspired framework for dynamic 3D scene understanding and reconstruction.<n>Our approach maintains a persistent memory bank of static scenes, enabling efficient spatial knowledge storage and rapid retrieval.
  arXiv  Detail & Related papers  (2026-01-13T03:09:35Z)
• ChangingGrounding: 3D Visual Grounding in Changing Scenes [92.00984845186679]
  Real-world robots localize objects from natural-language instructions while scenes around them keep changing.<n>Most of the existing 3D visual grounding (3DVG) method still assumes a reconstructed and up-to-date point cloud.<n>We introduce ChangingGrounding, the first benchmark that explicitly measures how well an agent can exploit past observations.
  arXiv  Detail & Related papers  (2025-10-16T17:59:16Z)
• Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft [45.363427511806385]
  Memory Forcing is a learning framework that pairs training protocols with a geometry-indexed spatial memory.<n>We show that Memory Forcing achieves superior long-term spatial consistency and generative quality across diverse environments.
  arXiv  Detail & Related papers  (2025-10-03T17:35:16Z)
• 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning [65.40458559619303]
  We propose 3D-Mem, a novel 3D scene memory framework for embodied agents.<n>3D-Mem employs informative multi-view images, termed Memory Snapshots, to represent the scene.<n>It further integrates frontier-based exploration by introducing Frontier Snapshots-glimpses of unexplored areas-enabling agents to make informed decisions.
  arXiv  Detail & Related papers  (2024-11-23T09:57:43Z)
• DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation [24.95545399754476]
  DynaMem is a new approach to open-world mobile manipulation that uses a dynamic work-semantic memory to represent a robot's environment.<n>Powered by DynaMem, our robots can explore novel environments, search for objects not found in memory, and continuously update the memory as objects move, appear, or disappear in the scene.
  arXiv  Detail & Related papers  (2024-11-07T18:59:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.