Uncertainty-Driven Replay Memory for Reinforcement Learning

• URL: http://arxiv.org/abs/2608.29860v1
• Date: Sun, 30 Aug 2026 15:41:57 GMT
• Title: Uncertainty-Driven Replay Memory for Reinforcement Learning
• Abstract summary: Uncertainty estimation provides promising capabilities for reinforcement learning (RL) agents.<n>We propose a novel formulation of the experience replay buffer commonly used in RL that we call uncertainty-driven replay memory (UDRM)
• Score: 43.273482638415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Uncertainty estimation provides promising capabilities for reinforcement learning (RL) agents. Notably, estimating uncertainty can reduce the training time and enable agents to obtain greater rewards over time by exploiting information related to whether an action would facilitate exploration of portions of an environment that are well-known versus those that are relatively unknown. In this work, we propose a novel formulation of the experience replay buffer commonly used in RL that we call uncertainty-driven replay memory (UDRM), which entails an update scheme for internally stored memories based on uncertainty estimates obtained by an RL model during training. In contrast to existing forms of RL, which typically use temporal difference error or the distribution of transitions to update the replay memory buffer and train RL controllers, our scheme biases the memory buffer to store more uncertain transitions that will improve an RL agent's generalization throughout training. Experimental results demonstrate that our proposed uncertainty-aware replay buffer enables an RL agent to obtain higher rewards during training compared to other existing uncertainty-aware RL frameworks.

Related papers

• Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training [11.074386515074446]
  "Memorization-guided Data Reuse" is a training paradigm that adaptively determines when and how data should be reused.<n>Preliminary experiments reveal a consistent memorization-driven regime.
  arXiv  Detail & Related papers  (2026-07-06T11:57:32Z)
• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning [19.683904518981624]
  ReRULE is an off-policy replay enhancement for reinforcement unlearning.<n>We show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE.
  arXiv  Detail & Related papers  (2026-06-13T14:52:40Z)
• Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs [59.62441340813425]
  We study whether reinforcement learning can improve direct recall of parametric knowledge.<n>Across three model families and multiple factual QA benchmarks, RL yields 27% average relative gains.<n>Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts.
  arXiv  Detail & Related papers  (2026-05-08T02:40:12Z)
• Efficient RL Training for LLMs with Experience Replay [33.7477995016327]
  We show that a well-designed replay buffer can drastically reduce inference compute without degrading.<n>We show that a well-designed replay buffer can drastically reduce inference compute without degrading.
  arXiv  Detail & Related papers  (2026-04-09T18:56:12Z)
• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• FIRE: A Failure-Adaptive Reinforcement Learning Framework for Edge Computing Migrations [54.34189781923818]
  FIRE is a framework that adapts to rare events by training a RL policy in an edge computing digital twin environment.<n>We propose ImRE, an importance sampling-based Q-learning algorithm, which samples rare events proportionally to their impact on the value function.<n>We show that FIRE reduces costs compared to vanilla RL and the greedy baseline in the event of failures.
  arXiv  Detail & Related papers  (2022-09-28T19:49:39Z)
• Low-Precision Reinforcement Learning [63.930246183244705]
  Low-precision training has become a popular approach to reduce computation time, memory footprint, and energy consumption in supervised learning. In this paper we consider continuous control with the state-of-the-art SAC agent and demonstrate that a na"ive adaptation of low-precision methods from supervised learning fails.
  arXiv  Detail & Related papers  (2021-02-26T16:16:28Z)
• Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning [17.3794999533024]
  We show that deep RL appears to struggle in the presence of extraneous data. Recent works have shown that the performance of Deep Q-Network (DQN) degrades when its replay memory becomes too large. We re-examine the motivation for sampling uniformly over a replay memory, and find that it may be flawed when using function approximation.
  arXiv  Detail & Related papers  (2021-02-22T19:29:18Z)
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning [67.34810824996887]
  Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. We propose Iterated Relearning (ITER) to improve generalisation of deep RL agents.
  arXiv  Detail & Related papers  (2020-06-10T13:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.