$μ$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models

• URL: http://arxiv.org/abs/2606.12497v1
• Date: Wed, 10 Jun 2026 13:26:40 GMT
• Title: $μ$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models
• Abstract summary: We present a controlled isolation study of recurrence in a strong pretrained VLA backbone.<n>Our formulation augments the transformer with a small set of learnable memory tokens carried across timesteps.<n>We interpret these results as a calibration of the capability envelope of minimal in-backbone recurrence.
• Score: 44.66763491853812
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language-action (VLA) models predict chunks of future actions from the current observation, an assumption that fails under partial observability, where decisions depend on information no longer visible. Existing memory-augmented VLAs simultaneously introduce recurrence, retrieval, compression modules, auxiliary objectives, hierarchical memory, or task-specific architectural changes, so the contribution of recurrence itself remains entangled with surrounding machinery. We present a controlled isolation study of recurrence in a strong pretrained VLA backbone. Our formulation augments the transformer with a small set of learnable memory tokens carried across timesteps and updated through self-attention, trained end to end with truncated backpropagation through time, with no auxiliary losses and no architectural changes. We instantiate this as $μ$VLA, a family of OpenVLA-OFT variants parameterized by memory width m, TBPTT length K, and the memory update rule (cross-step gradients or a detached EMA), so that recurrence is the only varying factor. On MIKASA-Robo, $μ$VLA improves average success rate on five training tasks from 0.42 to 0.84 at the strongest setting and reaches 0.23 on held-out tasks with the same memory structure versus 0.07 for the memoryless baseline. On tasks requiring different memory structure, performance remains near baseline. On LIBERO, the strongest recurrent variant achieves 96.2% average success, indicating no regression under full observability. We interpret these results as a calibration of the capability envelope of minimal in-backbone recurrence, identifying the regime in which it is sufficient and the regime where additional memory structure is required. Demos and videos can be found in https://avanturist322.github.io/mu-vla/.

Related papers

• Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloads [0.0]
  Analytical models of peak VRAM consumption for LLM inference decompose memory into weight-storage, KV-cache, and activation terms.<n>We evaluate this decomposition empirically within a strictly scoped measurement study.<n>We find no justification for deploying complex predictive VRAM models in highly quantized, weight-dominated regimes.
  arXiv  Detail & Related papers  (2026-08-15T08:39:43Z)
• InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories [6.141047838371132]
  InduceKV is a retrieval-based method that stores each selected training prefix as an attention-ready memory entry.<n>We show that InduceKV consistently improves over PEFT, MoE, replay, and prompt-retrieval baselines under matched memory budgets.
  arXiv  Detail & Related papers  (2026-07-02T10:45:28Z)
• Multi-Head Recurrent Memory Agents [6.282017902182834]
  Multi-Head Recurrent Memory partitions memory into independent heads governed by a stage-wise select-then-update strategy.<n>As a lightweight instantiation, we introduce Least-Recently-Updated MHM (MHM-LRU), which guarantees uniform head utilization with zero additional token overhead.
  arXiv  Detail & Related papers  (2026-07-01T22:38:54Z)
• Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks [69.19366746169906]
  We present PRISM, a transformer-based architecture for visuomotor policies to use short-term memory.<n>Piracy filters retrieved information to suppress irrelevant details, improving performance.<n> hierarchical architecture compresses local information into compact tokens, improving its compute and memory footprint.
  arXiv  Detail & Related papers  (2026-06-15T03:49:23Z)
• EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents [80.59925936278162]
  We present EvolveMem, a self-evolving memory architecture that exposes its full retrieval configuration as a structured action space optimized by an LLM-powered diagnosis module.<n>This closedloop selfevolution realizes an AutoResearch process: the system autonomously conducts iterative research cycles on its own architecture, replacing manual configuration tuning.
  arXiv  Detail & Related papers  (2026-05-13T17:12:44Z)
• HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation [2.9231828959903474]
  Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance.<n>We show that this failure is not resolved by extending context length alone in the current reactive execution setting.<n>We present HELM, a model-agnostic framework that addresses these deficiencies with three components.
  arXiv  Detail & Related papers  (2026-04-20T19:57:35Z)
• ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries [45.23935281952228]
  We introduce ReMem-VLA, a vision-language-action (VLA) model equipped with two sets of learnable queries.<n>These queries are trained end-to-end to aggregate and maintain relevant context over time.<n>We demonstrate that ReMem-VLA exhibits strong memory capabilities across multiple dimensions.
  arXiv  Detail & Related papers  (2026-03-13T12:38:42Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• Memory Caching: RNNs with Growing Memory [56.25483647131372]
  We introduce Memory Caching (MC), a technique that enhances recurrent models by caching checkpoints of memory states (a.k.a. hidden states)<n>We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules.<n>The results indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.
  arXiv  Detail & Related papers  (2026-02-27T18:53:41Z)
• KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning [8.216400469571084]
  We propose KEEP, a KV-cache-centric memory management system for efficient embodied planning.<n>KEEP features 3 key innovations: (1) a Static-Dynamic Memory Construction algorithm that reduces KV cache recomputation by mixed-granularity memory group; (2) a Multi-hop Memory Re-computation algorithm that dynamically identifies important cross-attention among different memory groups; and (3) a Layer-balanced Memory Loading that eliminates unbalanced KV cache loading and cross-attention across different layers.
  arXiv  Detail & Related papers  (2026-02-27T01:48:07Z)
• NextMem: Towards Latent Factual Memory for LLM-based Agents [58.35585202907478]
  NextMem is a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory.<n>Extensive experiments demonstrate that NextMem achieves superior performance.
  arXiv  Detail & Related papers  (2026-02-26T14:35:27Z)
• Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoning [38.78160379823724]
  Recurrent-Depth VLA (RD-VLA) is an architecture that achieves computational adaptivity via latent iterative refinement rather than explicit token generation.<n>RD-VLA employs a recurrent, weight-tied action head that supports arbitrary inference depth with a constant memory footprint.
  arXiv  Detail & Related papers  (2026-02-08T07:21:01Z)
• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL [48.214881182054164]
  We propose ELMUR, a transformer architecture with structured external memory.<n>ELMUR extends effective horizons up to 100,000 times beyond the attention window.<n>It achieves a 100% success rate on a synthetic T-Maze task with corridors up to one million steps.
  arXiv  Detail & Related papers  (2025-10-08T15:50:34Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.