Worth Remembering: Surprise-Gated Robot Episodic Memory

• URL: http://arxiv.org/abs/2606.03787v3
• Date: Sat, 06 Jun 2026 15:58:25 GMT
• Title: Worth Remembering: Surprise-Gated Robot Episodic Memory
• Abstract summary: We present an approach to compute surprise in a semantically rich deployment-agnostic latent space provided by V-JEPA-2.<n>Using our gated episodic memory to augment 4D scene graph-based spatial memory, we show a consistent improvement over state-of-the-art benchmarks in robot question answering.
• Score: 12.101245779148316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robots solving generalist tasks need to be able to ground instructions in their past experience, since humans may refer to notable past events when giving a task (e.g., ``Take me to where the chemical spill happened yesterday''). Since memory limits make storing all past events infeasible, long-term robot memory must be selective, ideally retaining only those episodes with high utility for future tasks. However, future tasks are not typically given a priori for generalist robots. To select generically useful memories, we propose Bayesian surprise as a gating mechanism for memory formation. We present an approach to compute surprise in a semantically rich deployment-agnostic latent space provided by V-JEPA-2. Using our gated episodic memory to augment 4D scene graph-based spatial memory, we show a consistent improvement over state-of-the-art benchmarks in robot question answering, outperforming prior robot memory methods by $\geq12\%$ for temporal, spatial, and binary questions, and surpassing the performance of supervised and non-causal methods with an unsupervised causal method in event segmentation tasks.

Related papers

• GaussMemory: Task-Driven 3D Gaussian Scene Memory for Long-Horizon Robotic Manipulation [10.233658861023612]
  Long-horizon robotic manipulation relies on persistent spatial memory.<n>Existing 3D memory systems store observations using fixed, hand-crafted rules.<n>We propose a paradigm shift from passive storage to active, task-driven spatial memory.
  arXiv  Detail & Related papers  (2026-08-15T02:28:59Z)
• Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering [93.80043825360514]
  Embodied question answering (EQA) is traditionally evaluated under an episodic formulation, where agents solve each task independently and reset internal state between episodes.<n>We investigate how different memory architectures behave when EQA agents are evaluated sequentially, with multiple questions answered in the same scene while memory is carried forward across queries.<n>We find that simply preserving existing memory is often insufficient.<n>We highlight the necessity of structured, spatially grounded memory: architectures that map persistent visual observations onto metric 3D geometry preserve visual-semantic evidence in a coherent scene representation.
  arXiv  Detail & Related papers  (2026-07-23T17:50:45Z)
• Benchmarking Robot Memory Under Interference [0.0]
  We introduce RoboMME-Interference, a cross-session benchmark built on RoboMME.<n>For each query episode, we construct a session history using the query's relevant prior demonstration followed by a controlled number of unrelated sessions.<n>Running RoboMME's released memory-augmented $_0.5$ variants unmodified through this benchmark, we find that while perceptual memory variants improve success when given the history without any distractors, they decay strongly and steadily as unrelated sessions accumulate.
  arXiv  Detail & Related papers  (2026-06-21T05:06:16Z)
• Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation [1.5192151934554061]
  We introduce Compressed Action Memory Policy ( CAMP)<n> CAMP is based on the insight that a robot's own action history serves as a highly informative, self-supervised signal.<n>We evaluate CAMP across four real-robot setups and two novel simulation benchmarks.
  arXiv  Detail & Related papers  (2026-06-19T07:56:58Z)
• MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models [80.70528162709276]
  Temporal modeling is essential for robotic manipulation, as effective control requires memory of past interactions and imagination of future states.<n>We propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation.<n>Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks.
  arXiv  Detail & Related papers  (2026-06-08T17:59:53Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models [73.3883864595845]
  We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies.<n>MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory.<n>MEM enables robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich.
  arXiv  Detail & Related papers  (2026-03-04T00:03:02Z)
• MemGen: Weaving Generative Latent Memory for Self-Evolving Agents [57.1835920227202]
  We propose MemGen, a dynamic generative memory framework that equips agents with a human-esque cognitive faculty.<n>MemGen enables agents to recall and augment latent memory throughout reasoning, producing a tightly interwoven cycle of memory and cognition.
  arXiv  Detail & Related papers  (2025-09-29T12:33:13Z)
• MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation [59.31354761628506]
  Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it.<n>We propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation.<n>We evaluate it on 150+ simulation and real-world tasks across three robots.
  arXiv  Detail & Related papers  (2025-08-26T17:57:16Z)
• Semantic HELM: A Human-Readable Memory for Reinforcement Learning [9.746397419479445]
  We propose a novel memory mechanism that represents past events in human language. We train our memory mechanism on a set of partially observable environments and find that it excels on tasks that require a memory component. Since our memory mechanism is human-readable, we can peek at an agent's memory and check whether crucial pieces of information have been stored.
  arXiv  Detail & Related papers  (2023-06-15T17:47:31Z)
• Evaluating Long-Term Memory in 3D Mazes [10.224858246626171]
  Memory Maze is a 3D domain of randomized mazes designed for evaluating long-term memory in agents. Unlike existing benchmarks, Memory Maze measures long-term memory separate from confounding agent abilities. We find that current algorithms benefit from training with truncated backpropagation through time and succeed on small mazes, but fall short of human performance on the large mazes.
  arXiv  Detail & Related papers  (2022-10-24T16:32:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.