MessyMem: Learning-from-Doing Memory for Mobile Manipulation

• URL: http://arxiv.org/abs/2609.15976v2
• Date: Tue, 15 Sep 2026 17:45:12 GMT
• Title: MessyMem: Learning-from-Doing Memory for Mobile Manipulation
• Abstract summary: MessyMem is a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks.<n>We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks.
• Score: 12.915040804075282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the robot should reuse that knowledge rather than start each task from scratch. Yet today's robots often treat each task as new: compact scene representations omit interaction-derived knowledge, raw video histories are difficult to query, and VLM planners reason at inference time without persistently updating what the robot knows. We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks. It maintains a spatially grounded 3D scene graph of objects and locations, augments it with properties and outcomes learned through interaction, and links visual observations for fine-grained recall. We evaluate MessyMem in simulation and on a real mobile manipulator. In a continuous 25-task simulation spanning over 3 hours, MessyMem achieves 80.0% task progress, outperforming the strongest ablation by 14.8 percentage points and the strongest external baseline by 28.9 points, while retrieving task-relevant evidence from thousands of stored keyframes and over an hour into the past.

Related papers

• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation [1.5192151934554061]
  We introduce Compressed Action Memory Policy ( CAMP)<n> CAMP is based on the insight that a robot's own action history serves as a highly informative, self-supervised signal.<n>We evaluate CAMP across four real-robot setups and two novel simulation benchmarks.
  arXiv  Detail & Related papers  (2026-06-19T07:56:58Z)
• Worth Remembering: Surprise-Gated Robot Episodic Memory [12.101245779148316]
  We present an approach to compute surprise in a semantically rich deployment-agnostic latent space provided by V-JEPA-2.<n>Using our gated episodic memory to augment 4D scene graph-based spatial memory, we show a consistent improvement over state-of-the-art benchmarks in robot question answering.
  arXiv  Detail & Related papers  (2026-06-02T15:39:27Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models [73.3883864595845]
  We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies.<n>MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory.<n>MEM enables robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich.
  arXiv  Detail & Related papers  (2026-03-04T00:03:02Z)
• MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation [59.31354761628506]
  Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it.<n>We propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation.<n>We evaluate it on 150+ simulation and real-world tasks across three robots.
  arXiv  Detail & Related papers  (2025-08-26T17:57:16Z)
• Experience is the Best Teacher: Grounding VLMs for Robotics through Self-Generated Memory [16.53566194731369]
  This paper presents a framework that grounds vision-language models (VLMs) to robots by building a self-generated memory of real-world experiences.<n>In ExpTeach, the VLM autonomously plans actions, verifies outcomes, reflects on failures, and adapts robot behaviors in a closed loop.<n>In experiments, we show that reflection improves success rates from 36% to 84% on four challenging robotic tasks.
  arXiv  Detail & Related papers  (2025-07-22T15:48:49Z)
• Towards General Purpose Robots at Scale: Lifelong Learning and Learning to Use Memory [0.0]
  This thesis focuses on addressing two key challenges for robots operating over long time horizons: memory and lifelong learning.<n>First, we introduce t-DGR, a trajectory-based deep generative replay method that achieves state-of-the-art performance on Continual World benchmarks.<n>Second, we develop a framework that leverages human demonstrations to teach agents effective memory utilization.
  arXiv  Detail & Related papers  (2024-12-28T21:13:48Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Learning Object-Based State Estimators for Household Robots [11.055133590909097]
  We build object-based memory systems that operate on high-dimensional observations and hypotheses. We demonstrate the system's effectiveness in maintaining memory of dynamically changing objects in both simulated environment and real images.
  arXiv  Detail & Related papers  (2020-11-06T04:18:52Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.