When Does Non-Uniform Replay Matter in Reinforcement Learning?

• URL: http://arxiv.org/abs/2605.10236v3
• Date: Sun, 17 May 2026 21:09:20 GMT
• Title: When Does Non-Uniform Replay Matter in Reinforcement Learning?
• Abstract summary: We show that the effectiveness of non-uniform replay is governed by three factors: replay volume, the number of replayed transitions per environment step; expected recency, how recent sampled transitions are; and the entropy of the replay sampling distribution.<n>We adopt a simple Truncated Geometric replay that biases sampling toward recent experience while preserving high entropy and incurring negligible computational overhead.
• Score: 43.83763359517116
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern off-policy reinforcement learning algorithms often rely on simple uniform replay sampling and it remains unclear when and why non-uniform replay improves over this strong baseline. Across diverse RL settings, we show that the effectiveness of non-uniform replay is governed by three factors: replay volume, the number of replayed transitions per environment step; expected recency, how recent sampled transitions are; and the entropy of the replay sampling distribution. Our main contribution is clarifying when non-uniform replay is beneficial and providing practical guidance for replay design in modern off-policy RL. Namely, we find that non-uniform replay is most beneficial when replay volume is low, and that high-entropy sampling is important even at comparable expected recency. Motivated by these findings, we adopt a simple Truncated Geometric replay that biases sampling toward recent experience while preserving high entropy and incurring negligible computational overhead. Across large-scale parallel simulation, single-task, and multi-task settings, including three modern algorithms evaluated on five RL benchmark suites, this replay sampling strategy improves sample efficiency in low-volume regimes while remaining competitive when replay volume is high.

Related papers

• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning [19.683904518981624]
  ReRULE is an off-policy replay enhancement for reinforcement unlearning.<n>We show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE.
  arXiv  Detail & Related papers  (2026-06-13T14:52:40Z)
• Efficient RL Training for LLMs with Experience Replay [33.7477995016327]
  We show that a well-designed replay buffer can drastically reduce inference compute without degrading.<n>We show that a well-designed replay buffer can drastically reduce inference compute without degrading.
  arXiv  Detail & Related papers  (2026-04-09T18:56:12Z)
• Replay Can Provably Increase Forgetting [24.538643224479515]
  A critical challenge for continual learning is forgetting, where the performance on previously learned tasks decreases as new tasks are introduced.<n>One of the commonly used techniques to mitigate forgetting, sample replay, has been shown empirically to reduce forgetting.<n>We show that even in a noiseless setting, forgetting can be non-monotonic with respect to the number of replay samples.
  arXiv  Detail & Related papers  (2025-06-04T18:46:23Z)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training [71.16258800411696]
  Reinforcement learning (RL) is a critical component of large language model (LLM) post-training.<n>Existing on-policy algorithms used for post-training are inherently incompatible with the use of experience replay buffers.<n>We propose efficiently obtaining this benefit of replay buffers via Trajectory Balance with Asynchrony (TBA)
  arXiv  Detail & Related papers  (2025-03-24T17:51:39Z)
• Experience Replay with Random Reshuffling [3.6622737533847936]
  In supervised learning, it is common to shuffle the dataset every epoch and consume data sequentially, which is called random reshuffling (RR)<n>We propose sampling methods that extend RR to experience replay, both in uniform and prioritized settings.<n>We evaluate our sampling methods on Atari benchmarks, demonstrating their effectiveness in deep reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-04T04:37:22Z)
• Layerwise Proximal Replay: A Proximal Point Method for Online Continual Learning [22.00843101957619]
  In online continual learning, a neural network incrementally learns from a non-i.i.d. data stream. Our work demonstrates a limitation of this approach: neural networks trained with experience replay tend to have unstable optimization trajectories. We present Layerwise Proximal Replay (LPR), which balances learning from new and replay data while only allowing for gradual changes in the hidden activation of past data.
  arXiv  Detail & Related papers  (2024-02-14T19:34:28Z)
• Adiabatic replay for continual learning [138.7878582237908]
  generative replay spends an increasing amount of time just re-learning what is already known. We propose a replay-based CL strategy that we term adiabatic replay (AR) We verify experimentally that AR is superior to state-of-the-art deep generative replay using VAEs.
  arXiv  Detail & Related papers  (2023-03-23T10:18:06Z)
• Replay For Safety [51.11953997546418]
  In experience replay, past transitions are stored in a memory buffer and re-used during learning. We show that using an appropriate biased sampling scheme can allow us to achieve a emphsafe policy.
  arXiv  Detail & Related papers  (2021-12-08T11:10:57Z)
• Convergence Results For Q-Learning With Experience Replay [51.11953997546418]
  We provide a convergence rate guarantee, and discuss how it compares to the convergence of Q-learning depending on important parameters such as the frequency and number of iterations of replay. We also provide theoretical evidence showing when we might expect this to strictly improve performance, by introducing and analyzing a simple class of MDPs.
  arXiv  Detail & Related papers  (2021-12-08T10:22:49Z)
• Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning [17.3794999533024]
  We show that deep RL appears to struggle in the presence of extraneous data. Recent works have shown that the performance of Deep Q-Network (DQN) degrades when its replay memory becomes too large. We re-examine the motivation for sampling uniformly over a replay memory, and find that it may be flawed when using function approximation.
  arXiv  Detail & Related papers  (2021-02-22T19:29:18Z)
• Revisiting Fundamentals of Experience Replay [91.24213515992595]
  We present a systematic and extensive analysis of experience replay in Q-learning methods. We focus on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected.
  arXiv  Detail & Related papers  (2020-07-13T21:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.