MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

• URL: http://arxiv.org/abs/2606.20679v1
• Date: Sat, 13 Jun 2026 08:54:52 GMT
• Title: MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
• Abstract summary: We present MemoryVAM, an episodic memory mechanism for video-world-model policies.<n>We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap maps per-frame CLIP embeddings into compact memory tokens.<n>These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history.
• Score: 32.07039868974098
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Video-world-model policies learn action-relevant representations by predicting future observations. However, they condition on only a short observation window, which renders long-horizon manipulation non-Markovian when the correct action depends on earlier events that are no longer visible. We present MemoryVAM, an episodic memory mechanism for video-world-model policies. We employ a Recap-Cue (RC) module, in which a Perceiver-based Recap Compressor maps per-frame CLIP embeddings into compact memory tokens, and a lightweight Cue Gate estimates task completion from memory and language. These tokens are injected into both the video backbone and the action decoder, aligning policy imagination with episode progress and conditioning actions on history. Our model trains the memory module with video prediction, a delta-reconstruction auxiliary loss, and episode-boundary supervision, requiring no per-frame progress labels. The same mechanism applies to UNet and Diffusion Transformer (DiT) backbones by changing only the cross-attention injection interface. On LIBERO-Mem, our model improves average success from 5% to 42.5%. On real robots, it achieves 78.3% success on counting tasks, 80.0% on spatial recall, and 75.0% on sequential tracking. Project page: https://MemoryVAM.github.io/

Related papers

• MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation [36.81053166644513]
  We present textbfMEMOBench, a benchmark for process level memory evaluation in robotic manipulation.<n>MEMOBench includes 30 history dependent tasks, 1,500 expert demonstrations, and 4,200 executable checkpoint instances from 84 templates.
  arXiv  Detail & Related papers  (2026-09-07T05:06:20Z)
• PonderPounce: A Pretrained MLLM as an Episode Context Engine for Robot Control [25.920573627524945]
  PonderPounce reuses an MLLM's native causal context as robot memory.<n>Pounce receives the current observation, instruction, and proprioception directly.<n>On RoboMME with base-scale training data, PonderPounce reaches 60.83% with 9B and 50.04% with 0.8B under the same Pounce architecture and interface.
  arXiv  Detail & Related papers  (2026-08-25T06:24:36Z)
• Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation [1.5192151934554061]
  We introduce Compressed Action Memory Policy ( CAMP)<n> CAMP is based on the insight that a robot's own action history serves as a highly informative, self-supervised signal.<n>We evaluate CAMP across four real-robot setups and two novel simulation benchmarks.
  arXiv  Detail & Related papers  (2026-06-19T07:56:58Z)
• MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models [80.70528162709276]
  Temporal modeling is essential for robotic manipulation, as effective control requires memory of past interactions and imagination of future states.<n>We propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation.<n>Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks.
  arXiv  Detail & Related papers  (2026-06-08T17:59:53Z)
• Echo-Memory: A Controlled Study of Memory in Action World Models [58.77172260403133]
  We present a controlled study of memory mechanisms in action-conditioned world models.<n>Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences.<n>We use Echo-Memory to fix the action-to-video interface and varies only how history is stored and read by the generator.
  arXiv  Detail & Related papers  (2026-06-08T17:54:10Z)
• VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory [31.464584758455356]
  VPWEM is a non-Markovian visuomotor policy equipped with working and episodic memories.<n>It exploits both short-term and episode-wide information for action generation with nearly constant memory and computation per step.
  arXiv  Detail & Related papers  (2026-03-05T07:52:50Z)
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models [73.3883864595845]
  We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies.<n>MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory.<n>MEM enables robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich.
  arXiv  Detail & Related papers  (2026-03-04T00:03:02Z)
• MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation [59.31354761628506]
  Temporal context is essential for robotic manipulation because such tasks are inherently non-Markovian, yet mainstream VLA models typically overlook it.<n>We propose MemoryVLA, a Cognition-Memory-Action framework for long-horizon robotic manipulation.<n>We evaluate it on 150+ simulation and real-world tasks across three robots.
  arXiv  Detail & Related papers  (2025-08-26T17:57:16Z)
• Physical Autoregressive Model for Robotic Manipulation without Action Pretraining [65.8971623698511]
  We build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR)<n>PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining.<n>Experiments on the ManiSkill benchmark show that PAR achieves a 100% success rate on the PushCube task.
  arXiv  Detail & Related papers  (2025-08-13T13:54:51Z)
• Token Turing Machines [53.22971546637947]
  Token Turing Machines (TTM) is a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history.
  arXiv  Detail & Related papers  (2022-11-16T18:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.