Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2606.11797v1
• Date: Wed, 10 Jun 2026 08:30:04 GMT
• Title: Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Deep Reinforcement Learning
• Abstract summary: Non-stationary Reinforcement Learning (NSRL) deals with adapting state-of-the-art RL methods to deal with changing environments.<n>This work develops emphSpace-sampled Value Decay as an explicit forgetting mechanism for value-based deep RL architectures.
• Score: 6.978294051213784
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Studies on rodents such as mice have shown the capabilities to adapt their behavior when dealing with changing parameters (``drift'') of the environment even if no information about change is provided (uncertainty) -- a behavior that can be modeled by forgetting mechanisms. Non-stationary Reinforcement Learning (NSRL) deals with adapting state-of-the-art RL methods to deal with changing environments: these however usually require (partially) perfect information about the drift such as ``task IDs'' or ``context''. To mitigate the effects of drift, this work develops \emph{Space-sampled Value Decay} as an explicit forgetting mechanism for value-based deep RL architectures as a simple yet effective approach. In particular we demonstrate and discuss positive effects but also limitations in achieved returns for modifications of Deep Q-networks (DQN) and Soft Actor-Critic (SAC) when evaluated on non-stationary environments.

Related papers

• Adaptive Undulatory Locomotion of Snake-like Robots in Dynamic Viscous Environments via Deep Reinforcement Learning [2.024925013349319]
  This paper demonstrates how deep reinforcement learning (DRL) enables adaptive locomotion of snake-like robots in dynamically changing environments.<n>By employing an asymmetric actor-critic framework, a teacher policy trained using privileged information available only in the physics simulator distills its knowledge into a student policy.<n>The findings establish that implicit environment inference via privileged information distillation is an effective approach to bypass the constraints of classical models under unpredictable fluid dynamics.
  arXiv  Detail & Related papers  (2026-07-24T04:14:36Z)
• Learning Context-Aware Neural ODE Dynamics for Adaptive Robotic Control [55.01272784312405]
  We propose a context-aware dynamics model based on neural ordinary differential equations.<n>We validate the approach across diverse robotic platforms, including a quadrotor in simulation, as well as a Sphero BOLT robot and a Fanuc manipulator in real-world experiments.
  arXiv  Detail & Related papers  (2026-06-13T20:55:46Z)
• The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning [62.72522261010872]
  Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity.<n>We study the plasticity loss problem from the theoretical perspective of network optimization.<n>We propose Sample Weight Decay -- a lightweight method to restore gradient magnitude.
  arXiv  Detail & Related papers  (2026-04-02T11:29:46Z)
• Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics [0.5745796568988237]
  In this paper, we propose a lightweight online adaptation framework forReinforcement learning (RL) based on Reservoir Computing.<n> Specifically, we integrate an Echo State Networks (ESNs) as an adaptation module that encodes recent observation histories into a latent context representation.<n>We evaluate the proposed method on CartPole and HalfCheetah tasks with severe and abrupt environment changes.
  arXiv  Detail & Related papers  (2026-02-06T02:51:01Z)
• Adapting to Fragmented and Evolving Data: A Fisher Information Perspective [0.0]
  FADE is a lightweight framework for robust learning under dynamic environments.<n>It employs a shift-aware regularization mechanism anchored in Fisher information geometry.<n>FADE operates online with fixed memory and no access to target labels.
  arXiv  Detail & Related papers  (2025-07-25T06:50:09Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• Dynamic Reinforcement Learning for Actors [0.0]
  Dynamic Reinforcement Learning (Dynamic RL) directly controls system dynamics, instead of the actor (action-generating neural network) outputs at each moment.<n>Actor is initially designed to generate chaotic dynamics through the loop with its environment.<n>Dynamic RL controls global system dynamics using a local index called "sensitivity"
  arXiv  Detail & Related papers  (2025-02-14T14:50:05Z)
• datadriftR: An R Package for Concept Drift Detection in Predictive Models [0.0]
  This paper introduces drifter, an R package designed to detect concept drift.<n>It proposes a novel method called Profile Drift Detection (PDD) that enables both drift detection and an enhanced understanding of the cause behind the drift.
  arXiv  Detail & Related papers  (2024-12-15T20:59:49Z)
• Exploring Test-Time Adaptation for Object Detection in Continually Changing Environments [20.307151769610087]
  Continual Test-Time Adaptation (CTTA) has emerged as a promising technique to gradually adapt a source-trained model to continually changing target domains.<n>We present AMROD, featuring three core components, to tackle these challenges for detection models in CTTA scenarios.<n>We demonstrate the effectiveness of AMROD on four CTTA object detection tasks, where AMROD outperforms existing methods.
  arXiv  Detail & Related papers  (2024-06-24T08:30:03Z)
• Analyzing Adversarial Inputs in Deep Reinforcement Learning [53.3760591018817]
  We present a comprehensive analysis of the characterization of adversarial inputs, through the lens of formal verification. We introduce a novel metric, the Adversarial Rate, to classify models based on their susceptibility to such perturbations. Our analysis empirically demonstrates how adversarial inputs can affect the safety of a given DRL system with respect to such perturbations.
  arXiv  Detail & Related papers  (2024-02-07T21:58:40Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)
• Few-shot Quality-Diversity Optimization [50.337225556491774]
  Quality-Diversity (QD) optimization has been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. We show that, given examples from a task distribution, information about the paths taken by optimization in parameter space can be leveraged to build a prior population, which when used to initialize QD methods in unseen environments, allows for few-shot adaptation. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimization in these environments.
  arXiv  Detail & Related papers  (2021-09-14T17:12:20Z)
• Attribute-Guided Adversarial Training for Robustness to Natural Perturbations [64.35805267250682]
  We propose an adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to the attributes-space. Our approach enables deep neural networks to be robust against a wide range of naturally occurring perturbations.
  arXiv  Detail & Related papers  (2020-12-03T10:17:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.