ARROW: Augmented Replay for RObust World models

• URL: http://arxiv.org/abs/2603.11395v1
• Date: Thu, 12 Mar 2026 00:15:11 GMT
• Title: ARROW: Augmented Replay for RObust World models
• Abstract summary: Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones.<n>Most existing approaches rely on model-free methods with replay buffers to mitigate catastrophic forgetting.<n>We present ARROW, a model-based continual RL algorithm that extends DreamerV3 with a memory-efficient, distribution-matching replay buffer.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual reinforcement learning challenges agents to acquire new skills while retaining previously learned ones with the goal of improving performance in both past and future tasks. Most existing approaches rely on model-free methods with replay buffers to mitigate catastrophic forgetting; however, these solutions often face significant scalability challenges due to large memory demands. Drawing inspiration from neuroscience, where the brain replays experiences to a predictive World Model rather than directly to the policy, we present ARROW (Augmented Replay for RObust World models), a model-based continual RL algorithm that extends DreamerV3 with a memory-efficient, distribution-matching replay buffer. Unlike standard fixed-size FIFO buffers, ARROW maintains two complementary buffers: a short-term buffer for recent experiences and a long-term buffer that preserves task diversity through intelligent sampling. We evaluate ARROW on two challenging continual RL settings: Tasks without shared structure (Atari), and tasks with shared structure, where knowledge transfer is possible (Procgen CoinRun variants). Compared to model-free and model-based baselines with replay buffers of the same-size, ARROW demonstrates substantially less forgetting on tasks without shared structure, while maintaining comparable forward transfer. Our findings highlight the potential of model-based RL and bio-inspired approaches for continual reinforcement learning, warranting further research.

Related papers

• Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers [7.6344611819427035]
  Multiple Embedding Replay Selection, MERS, replaces the buffer selection module with a graph-based approach.<n>We show consistent improvements over SOTA selection strategies across a range of continual learning algorithms.<n>MERS is a practical, drop-in enhancement for replay-based continual learning.
  arXiv  Detail & Related papers  (2026-04-09T15:09:02Z)
• DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay [57.80564154223355]
  Existing experience replay methods address this by reusing accurate samples for direct policy updates.<n>We argue that historical data should prioritize sustaining diversity rather than simply reinforcing accuracy.<n>We propose Dynamic Jensen-Shannon Replay (LEPJR), a simple yet effective regularization framework.
  arXiv  Detail & Related papers  (2026-03-17T06:20:56Z)
• Bagging-Based Model Merging for Robust General Text Embeddings [73.51674133699196]
  General-purpose text embedding models underpin a wide range of NLP and information retrieval applications.<n>We present a systematic study of multi-task training for text embeddings from two perspectives: data scheduling and model merging.<n>We propose Bagging-based rObust mOdel Merging (BOOM), which trains multiple embedding models on sampled subsets and merges them into a single model.
  arXiv  Detail & Related papers  (2026-02-05T15:45:08Z)
• Scalable Offline Model-Based RL with Action Chunks [60.80151356018376]
  We study whether model-based reinforcement learning can provide a scalable recipe for tackling complex, long-horizon tasks in offline RL.<n>We call this recipe textbfModel-Based RL with Action Chunks (MAC).<n>We show that MAC achieves the best performance among offline model-based RL algorithms, especially on challenging long-horizon tasks.
  arXiv  Detail & Related papers  (2025-12-08T23:26:29Z)
• Buffer replay enhances the robustness of multimodal learning under missing-modality [9.512378886218395]
  We introduce REplay Prompting (REP), which builds modality-wise buffers and replays them in deeper layers to mitigate information loss as network depth increases.<n>Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios.<n>These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.
  arXiv  Detail & Related papers  (2025-11-28T10:55:31Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Intention-Conditioned Flow Occupancy Models [80.42634994902858]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training [71.16258800411696]
  Reinforcement learning (RL) is a critical component of large language model (LLM) post-training.<n>Existing on-policy algorithms used for post-training are inherently incompatible with the use of experience replay buffers.<n>We propose efficiently obtaining this benefit of replay buffers via Trajectory Balance with Asynchrony (TBA)
  arXiv  Detail & Related papers  (2025-03-24T17:51:39Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Augmenting Replay in World Models for Continual Reinforcement Learning [0.0]
  Continual RL requires an agent to learn new tasks without forgetting previous ones, while improving on both past and future tasks. The most common approaches use model-free algorithms and replay buffers to mitigate catastrophic forgetting. We introduce WMAR (World Models with Augmented Replay), a model-based RL algorithm with a memory-efficient replay buffer.
  arXiv  Detail & Related papers  (2024-01-30T00:48:26Z)
• Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2 [5.239932780277599]
  Generative replay (GR) is a biologically-inspired replay mechanism that augments learning experiences with self-labelled examples. We present a version of GR for LRL that satisfies two desideratas: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning.
  arXiv  Detail & Related papers  (2022-08-09T22:00:28Z)
• Stable and expressive recurrent vision models [12.578121388491764]
  "contractor recurrent back-propagation" (C-RBP) is a new learning algorithm that achieves constant O(1) memory-complexity with steps of recurrent processing. C-RBP is a general-purpose learning algorithm for any application that can benefit from recurrent dynamics.
  arXiv  Detail & Related papers  (2020-05-22T19:31:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.