Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks

• URL: http://arxiv.org/abs/2606.16178v1
• Date: Mon, 15 Jun 2026 03:49:23 GMT
• Title: Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks
• Abstract summary: We present PRISM, a transformer-based architecture for visuomotor policies to use short-term memory.<n>Piracy filters retrieved information to suppress irrelevant details, improving performance.<n> hierarchical architecture compresses local information into compact tokens, improving its compute and memory footprint.
• Score: 69.19366746169906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many robotic tasks require short-term memory, whether it's retrieving an object that's no longer visible or turning off an appliance after a set period. Yet, most visuomotor policies trained via imitation learning rely only on immediate sensory input without using past experiences to guide decisions. We present PRISM, a transformer-based architecture for visuomotor policies to effectively use short-term memory via two key components: (i) gated attention, which filters retrieved information to suppress irrelevant details, improving performance by reducing the spurious correlations between the history and current action prediction, (ii) a hierarchical architecture that first compresses local information into compact tokens and then integrates them to capture temporally extended dependencies, improving its compute and memory footprint. Together, these mechanisms enable us to scale short-term memory in visuomotor policies for up to two minutes. To systematically evaluate memory in visuomotor control, we introduce ReMemBench -- a benchmark of eight diverse household manipulation tasks spanning four categories of short-term memory -- designed to foster general memory mechanisms rather than siloed, task-specific solutions. PRISM consistently outperforms prior works, including recurrent architectures, transformers, and their variants -- achieving an absolute improvement of 5%--12% over the strongest baseline. On the RoboCasa and LIBERO benchmarks, it achieves absolute improvements of 11%--15% over its no-memory variant and fine-tuned Vision-Language-Action baselines such as GR00T-N1-3B and OpenVLA, despite not leveraging any large-scale pretraining. Together, PRISM and ReMemBench establish a foundation for developing and evaluating short-term memory-augmented visuomotor policies that scale to long-horizon tasks. Additional materials are available at https://shahrutav.github.io/short-term-memory

Related papers

• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• Memory Retrieval in Visuomotor Policies for Long-Horizon Robot Control [33.5619212312672]
  General-purpose robots operating in partially observable environments, such as homes, require memory to support autonomy.<n>We introduce HALO, a visuomotor policy with an attention-based memory retrieval mechanism for long-horizon control.
  arXiv  Detail & Related papers  (2026-06-23T20:07:23Z)
• RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark [49.16825786282095]
  Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks.<n>Existing robotic memory benchmarks still lack multimodal annotations for memory formation, provide limited task coverage and structural complexity, and remain restricted to simulation without real-world evaluation.<n>We address this gap with RoboMemArena, a large-scale benchmark of 26 tasks, with average trajectory lengths exceeding 1,000 steps per task and 68.9% of subtasks being memory-dependent.
  arXiv  Detail & Related papers  (2026-05-11T17:54:49Z)
• MemoAct: Atkinson-Shiffrin-Inspired Memory-Augmented Visuomotor Policy for Robotic Manipulation [6.490934654648497]
  MemoAct is a hierarchical memory-based policy that leverages distinct memory tiers to tackle specific bottlenecks.<n>MemoAct achieves superior performance compared to both existing Markovian baselines and history-aware policies.
  arXiv  Detail & Related papers  (2026-03-19T05:02:43Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design [77.30163153176954]
  RMBench is a simulation benchmark comprising 9 manipulation tasks that span multiple levels of memory complexity.<n>Mem-0 is a modular manipulation policy with explicit memory components designed to support controlled ablation studies.<n>We identify memory-related limitations in existing policies and provide empirical insights into how architectural design choices influence memory performance.
  arXiv  Detail & Related papers  (2026-03-01T18:59:59Z)
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents [57.38404718635204]
  Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows.<n>Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components.<n>We propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy.
  arXiv  Detail & Related papers  (2026-01-05T08:24:16Z)
• Preference-Aware Memory Update for Long-Term LLM Agents [27.776042930733784]
  One of the key factors influencing the reasoning capabilities of LLM-based agents is their ability to leverage long-term memory.<n>We propose a Preference-Aware Memory Update Mechanism (PAMU) that enables dynamic and personalized memory refinement.
  arXiv  Detail & Related papers  (2025-10-10T06:49:35Z)
• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL [48.214881182054164]
  We propose ELMUR, a transformer architecture with structured external memory.<n>ELMUR extends effective horizons up to 100,000 times beyond the attention window.<n>It achieves a 100% success rate on a synthetic T-Maze task with corridors up to one million steps.
  arXiv  Detail & Related papers  (2025-10-08T15:50:34Z)
• MeMSVD: Long-Range Temporal Structure Capturing Using Incremental SVD [27.472705540825316]
  This paper is on long-term video understanding where the goal is to recognise human actions over long temporal windows (up to minutes long) We propose an alternative to attention-based schemes which is based on a low-rank approximation of the memory obtained using Singular Value Decomposition. Our scheme has two advantages: (a) it reduces complexity by more than an order of magnitude, and (b) it is amenable to an efficient implementation for the calculation of the memory bases.
  arXiv  Detail & Related papers  (2024-06-11T12:03:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.