MemER: Scaling Up Memory for Robot Control via Experience Retrieval

• URL: http://arxiv.org/abs/2510.20328v1
• Date: Thu, 23 Oct 2025 08:26:17 GMT
• Title: MemER: Scaling Up Memory for Robot Control via Experience Retrieval
• Abstract summary: Humans routinely rely on memory to tasks, yet most robot policies lack this capability.<n>We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevants from its experience.<n>Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory.
• Score: 46.5398413633767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and $\pi_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.

Related papers

• $τ_0$-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation [51.05516457377339]
  Long-horizon robot manipulation requires a robot to both execute individual skills reliably and sequence them coherently over extended tasks.<n>We introduce $_0$-VLA, a hierarchical robot foundation model that formulates high-level subtask generation as a compute-scalable inference problem.<n>A low-level policy then executes the generated subtask across multiple robot embodiments.
  arXiv  Detail & Related papers  (2026-08-17T17:59:11Z)
• Remember what you did?: Learning Behavioral Memories for Partially Observable Object Manipulation [1.5192151934554061]
  We introduce Compressed Action Memory Policy ( CAMP)<n> CAMP is based on the insight that a robot's own action history serves as a highly informative, self-supervised signal.<n>We evaluate CAMP across four real-robot setups and two novel simulation benchmarks.
  arXiv  Detail & Related papers  (2026-06-19T07:56:58Z)
• Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks [69.19366746169906]
  We present PRISM, a transformer-based architecture for visuomotor policies to use short-term memory.<n>Piracy filters retrieved information to suppress irrelevant details, improving performance.<n> hierarchical architecture compresses local information into compact tokens, improving its compute and memory footprint.
  arXiv  Detail & Related papers  (2026-06-15T03:49:23Z)
• MemoAct: Atkinson-Shiffrin-Inspired Memory-Augmented Visuomotor Policy for Robotic Manipulation [6.490934654648497]
  MemoAct is a hierarchical memory-based policy that leverages distinct memory tiers to tackle specific bottlenecks.<n>MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies.
  arXiv  Detail & Related papers  (2026-03-19T05:02:43Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• MEM: Multi-Scale Embodied Memory for Vision Language Action Models [73.3883864595845]
  We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies.<n>MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory.<n>MEM enables robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich.
  arXiv  Detail & Related papers  (2026-03-04T00:03:02Z)
• Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining [56.62125584296097]
  Keyframe-Chaining VLA is a framework that extracts and links key historical frames to model long-horizon dependencies.<n>We design a progress-aware mechanism that dynamically retrieves historical frames based on their temporal relevance to the current execution phase.<n>We introduce a suite of four Non-Markovian manipulation tasks built upon the ManiSkill simulator to measure task success rates.
  arXiv  Detail & Related papers  (2026-03-02T05:26:29Z)
• RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design [77.30163153176954]
  RMBench is a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity.<n>Mem-0 is a modular manipulation policy with explicit memory components designed to support controlled ablation studies.<n>We identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance.
  arXiv  Detail & Related papers  (2026-03-01T18:59:59Z)
• From Code to Action: Hierarchical Learning of Diffusion-VLM Policies [8.0703783175731]
  Imitation learning for robotic manipulation often suffers from limited generalization and data scarcity.<n>In this work, we introduce a hierarchical framework that leverages code-generating vision-language models (VLMs)<n>We find that this design enables interpretable policy decomposition, improves generalization when compared to flat policies and enables separate evaluation of high-level planning and low-level control.
  arXiv  Detail & Related papers  (2025-09-29T15:22:18Z)
• HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation [54.03004125910057]
  We show that hierarchical vision-language-action models can be more effective in utilizing off-domain data than standard monolithic VLA models.<n>We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios.
  arXiv  Detail & Related papers  (2025-02-08T07:50:22Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• Robot Learning of Mobile Manipulation with Reachability Behavior Priors [38.49783454634775]
  Mobile Manipulation (MM) systems are ideal candidates for taking up the role of a personal assistant in unstructured real-world environments. Among other challenges, MM requires effective coordination of the robot's embodiments for executing tasks that require both mobility and manipulation. We study the integration of robotic reachability priors in actor-critic RL methods for accelerating the learning of MM for reaching and fetching tasks.
  arXiv  Detail & Related papers  (2022-03-08T12:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.