Memory as Plans: World-Action Modeling with Memory-Grounded Planning

• URL: http://arxiv.org/abs/2609.11561v1
• Date: Thu, 10 Sep 2026 13:52:51 GMT
• Title: Memory as Plans: World-Action Modeling with Memory-Grounded Planning
• Abstract summary: MaP-WAM is a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution.<n>MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks.
• Score: 33.29460541345171
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mainstream robotic policies often adopt a Markovian formulation, but many complex real-world manipulation tasks are inherently non-Markovian, requiring long-horizon memory beyond the current observation. Existing memory mechanisms often rely on language summaries, growing visual windows, or their combinations, and may therefore lose fine-grained visual evidence or face a trade-off between history coverage and execution efficiency. We introduce MaP-WAM, a Memory-as-Plans framework that decomposes memory-dependent world-action modeling into memory-grounded planning and plan-conditioned execution, and uses long-term multimodal episodic context as planning-time evidence rather than repeatedly conditioning the executor on the full history. MaP-WAM represents memory as completed segment records containing language instructions and sparse visual context, and converts this episodic memory into compact plans comprising the next segment-level language plan and corresponding visual guidance. A World-Action-Progress (WAP) model executes each plan over an unknown duration by jointly predicting action chunks and corresponding execution progress at inference time, calibrating predicted progress through plan-observation alignment for adaptive segment transitions and closed-loop context updates. MaP-WAM keeps the executor context length fixed, while structured attention further enables key-value caching in both planning and execution. MaP-WAM achieves state-of-the-art performance on RMBench with an 83.3% success rate and attains 78.0% success on real-robot tasks, while maintaining approximately constant executor inference latency as task history grows.

Related papers

• CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning [46.278651660199664]
  Planning is a central capability that enables agents to decompose complex longhorizon tasks into manageable steps.<n>Testtime search and training-based methods improve planning but incur high inference costs.<n>Self-evolving memory instead accumulates reusable experience from agent interaction outcomes into an external memory bank.
  arXiv  Detail & Related papers  (2026-09-02T03:59:21Z)
• WorldScape Policy 2.0: Empowering Steerable World Action Modeling with Reasoning-Augmented Memory [50.19449886428936]
  We introduce WorldScape Policy 2.0, a controllable WAM with reasoning-augmented long short-term memory.<n>The retrieved history augments perception and autoregressively generated planning tokens, yielding an implicit subgoal condition for autonomous planning.<n>Experiments in both simulation and real-world platforms demonstrate superior capabilities in long-horizon autonomous planning, fine-grained instruction following, and in-context adaptation.
  arXiv  Detail & Related papers  (2026-07-21T08:25:37Z)
• MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning [27.103249755262212]
  Long-horizon robot planning requires memory of the embodied experience that makes future goals interpretable.<n>We formulate Embodied Action Memory (EAM) as the capability to form, maintain, and use such experience as a persistent memory state for later decisions.<n>MEMORA realizes EAM with a formation-consolidation-retrieval lifecycle and four typed stores.
  arXiv  Detail & Related papers  (2026-07-15T18:12:28Z)
• MemoryWAM: Efficient World Action Modeling with Persistent Memory [80.90899269062128]
  We introduce MemoryWAM, a world action model with efficient persistent memory.<n>A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context.<n>MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines in both simulation and the real world.
  arXiv  Detail & Related papers  (2026-06-18T17:59:51Z)
• MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios [33.8882826707344]
  MemGround is a rigorous long-term memory benchmark grounded in rich, gamified interactive scenarios.<n>To comprehensively quantify both memory utilization and behavioral trajectories, we propose a multi-dimensional metric suite.
  arXiv  Detail & Related papers  (2026-03-23T02:57:39Z)
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies [54.23445842621374]
  Memory is critical for long-horizon and history-dependent robotic manipulation.<n>Recent vision-language-action (VLA) models have begun to incorporate memory mechanisms.<n>We introduce RoboMME: a large-scale standardized benchmark for evaluating and advancing VLA models.
  arXiv  Detail & Related papers  (2026-03-04T21:59:32Z)
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents [57.38404718635204]
  Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows.<n>Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components.<n>We propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy.
  arXiv  Detail & Related papers  (2026-01-05T08:24:16Z)
• FindingDory: A Benchmark to Evaluate Memory in Embodied Agents [49.18498389833308]
  We introduce a new benchmark for long-range embodied tasks in the Habitat simulator.<n>This benchmark evaluates memory-based capabilities across 60 tasks requiring sustained engagement and contextual awareness.
  arXiv  Detail & Related papers  (2025-06-18T17:06:28Z)
• EPD: Long-term Memory Extraction, Context-awared Planning and Multi-iteration Decision @ EgoPlan Challenge ICML 2024 [50.89751993430737]
  We introduce a novel planning framework which comprises three stages: long-term memory Extraction, context-awared Planning, and multi-iteration Decision, named EPD. EPD achieves a planning accuracy of 53.85% over 1,584 egocentric task planning questions.
  arXiv  Detail & Related papers  (2024-07-28T15:14:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.