Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2508.09277v1
• Date: Tue, 12 Aug 2025 18:32:08 GMT
• Title: Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning
• Authors: Soumia Mehimeh,
• Abstract summary: We introduce DQInit, a method that adapts value function initialization to deep reinforcement learning.<n>DQInit reuses compact Q-values extracted from previously solved tasks as a transferable knowledge base.<n>It employs a knownness-based mechanism to softly integrate these transferred values into underexplored regions and gradually shift toward the agent's learned estimates.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Value function initialization (VFI) is an effective way to achieve a jumpstart in reinforcement learning (RL) by leveraging value estimates from prior tasks. While this approach is well established in tabular settings, extending it to deep reinforcement learning (DRL) poses challenges due to the continuous nature of the state-action space, the noisy approximations of neural networks, and the impracticality of storing all past models for reuse. In this work, we address these challenges and introduce DQInit, a method that adapts value function initialization to DRL. DQInit reuses compact tabular Q-values extracted from previously solved tasks as a transferable knowledge base. It employs a knownness-based mechanism to softly integrate these transferred values into underexplored regions and gradually shift toward the agent's learned estimates, avoiding the limitations of fixed time decay. Our approach offers a novel perspective on knowledge transfer in DRL by relying solely on value estimates rather than policies or demonstrations, effectively combining the strengths of jumpstart RL and policy distillation while mitigating their drawbacks. Experiments across multiple continuous control tasks demonstrate that DQInit consistently improves early learning efficiency, stability, and overall performance compared to standard initialization and existing transfer techniques.

Related papers

• IDER: IDempotent Experience Replay for Reliable Continual Learning [36.35747221992653]
  Catastrophic forgetting is a major challenge in continual learning (CL)<n>Existing uncertainty-aware CL methods suffer from high computational overhead and incompatibility with mainstream replay methods.<n>We propose idempotent experience replay (IDER), a novel approach based on the idempotent property where repeated function applications yield the same output.
  arXiv  Detail & Related papers  (2026-02-28T12:34:18Z)
• In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Prior [53.21550098214227]
  In-context reinforcement learning promises fast adaptation to unseen environments without parameter updates.<n>We introduce SPICE, a Bayesian ICRL method that learns a prior over Q-values via deep ensemble and updates this prior at test-time.<n>We prove that SPICE achieves regret-optimal behaviour in both bandits and finite-horizon MDPs, even when pretrained only on suboptimal trajectories.
  arXiv  Detail & Related papers  (2026-01-06T13:41:31Z)
• Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning [51.07663354001582]
  Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task.<n>We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches.<n>We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods.
  arXiv  Detail & Related papers  (2025-12-01T15:56:00Z)
• Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer [0.9558392439655014]
  We propose a novel deep Q-learning approach that introduces a regularization term to reduce positive correlations between feature representation of states.<n>We demonstrate the efficacy of our approach in improving transfer learning performance and thereby reducing computational overhead.
  arXiv  Detail & Related papers  (2025-09-29T15:44:35Z)
• EKPC: Elastic Knowledge Preservation and Compensation for Class-Incremental Learning [53.88000987041739]
  Class-Incremental Learning (CIL) aims to enable AI models to continuously learn from sequentially arriving data of different classes over time.<n>We propose the Elastic Knowledge Preservation and Compensation (EKPC) method, integrating Importance-aware importance Regularization (IPR) and Trainable Semantic Drift Compensation (TSDC) for CIL.
  arXiv  Detail & Related papers  (2025-06-14T05:19:58Z)
• CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning [6.738409533239947]
  Deep neural networks struggle with catastrophic forgetting when learning tasks sequentially.<n>Recent approaches constrain updates to subspaces using gradient projection.<n>We propose Conceptor-based gradient projection for Deep Continual Learning (CODE-CL)
  arXiv  Detail & Related papers  (2024-11-21T22:31:06Z)
• Temporal-Difference Variational Continual Learning [89.32940051152782]
  We propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations.<n>Our approach effectively mitigates Catastrophic Forgetting, outperforming strong Variational CL methods.
  arXiv  Detail & Related papers  (2024-10-10T10:58:41Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning [20.07425661382103]
  $Q$-Aided Conditional Supervised Learning combines stability of RCSL with the stitching capability of $Q$-functions. QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return.
  arXiv  Detail & Related papers  (2024-02-03T04:17:09Z)
• Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agents [17.96977778655143]
  reset method performs periodic resets of a portion or the entirety of a deep RL agent while preserving the replay buffer. We propose a new reset-based method that leverages deep ensemble learning to address the limitations of the vanilla reset method.
  arXiv  Detail & Related papers  (2023-10-31T08:59:39Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Conservative Q-Learning for Offline Reinforcement Learning [106.05582605650932]
  We show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
  arXiv  Detail & Related papers  (2020-06-08T17:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.