Energy-Based Transfer for Reinforcement Learning

• URL: http://arxiv.org/abs/2506.16590v1
• Date: Thu, 19 Jun 2025 20:25:52 GMT
• Title: Energy-Based Transfer for Reinforcement Learning
• Authors: Zeyun Deng, Jasorsi Ghosh, Fiona Xie, Yuzhe Lu, Katia Sycara, Joseph Campbell,
• Abstract summary: Reinforcement learning algorithms often suffer from poor sample efficiency.<n>We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance.
• Score: 3.731813802304468
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning algorithms often suffer from poor sample efficiency, making them challenging to apply in multi-task or continual learning settings. Efficiency can be improved by transferring knowledge from a previously trained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically show that energy scores reflect the teacher's state-visitation density and empirically demonstrate improved sample efficiency and performance across both single-task and multi-task settings.

Related papers

• FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning [57.4737157531239]
  Transfer Learning offers the potential to accelerate learning by transferring knowledge across tasks.<n>It faces critical challenges such as negative transfer, domain adaptation and inefficiency in selecting solid source policies.<n>In this work we challenge the key issues in TL to improve knowledge transfer, agents performance across tasks and reduce computational costs.
  arXiv  Detail & Related papers  (2025-07-27T22:21:53Z)
• Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multi-Task Learning Perspective [55.531894882776726]
  Multi-task learning methods suggest using auxiliary tasks to enhance a neural network's performance on a specific primary task.<n>Previous methods often select auxiliary tasks carefully but treat them as secondary during training.<n>We propose an uncertainty-based impartial learning method that ensures balanced training across all tasks.
  arXiv  Detail & Related papers  (2024-12-27T09:27:18Z)
• Auxiliary Reward Generation with Transition Distance Representation Learning [20.150691753213817]
  Reinforcement learning (RL) has shown its strength in challenging sequential decision-making problems. The reward function in RL is crucial to the learning performance, as it serves as a measure of the task completion degree. We propose a novel representation learning approach that can measure the transition distance'' between states.
  arXiv  Detail & Related papers  (2024-02-12T05:13:44Z)
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization [0.24475591916185496]
  In multi-task reinforcement learning, it is possible to improve the data efficiency of training agents by transferring knowledge from other different but related tasks. Traditional methods rely on Kullback-Leibler regularization to stabilize the transfer of knowledge from one task to the others. In this work, we explore the direction of replacing the Kullback-Leibler divergence with a novel Optimal transport-based regularization.
  arXiv  Detail & Related papers  (2023-09-27T12:06:34Z)
• Introspective Action Advising for Interpretable Transfer Learning [7.673465837624365]
  Transfer learning can be applied in deep reinforcement learning to accelerate the training of a policy in a target task. We propose an alternative approach to transfer learning between tasks based on action advising, in which a teacher trained in a source task actively guides a student's exploration in a target task.
  arXiv  Detail & Related papers  (2023-06-21T14:53:33Z)
• Efficient Multi-Task and Transfer Reinforcement Learning with Parameter-Compositional Framework [44.43196786555784]
  We investigate the potential of improving multi-task training and leveraging it for transferring in the reinforcement learning setting. We propose a transferring approach with a parameter-compositional formulation. Experimental results demonstrate that the proposed approach can have improved performance in the multi-task training stage.
  arXiv  Detail & Related papers  (2023-06-02T18:00:33Z)
• ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning [59.08197876733052]
  Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks. Sometimes, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, known as negative transfer. ForkMerge is a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights.
  arXiv  Detail & Related papers  (2023-01-30T02:27:02Z)
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning [13.36223726517518]
  This work proposes REPresentation And IN Transfer (REPAINT) algorithm for knowledge transfer in deep reinforcement learning. REPAINT not only transfers the representation of a pre-trained teacher policy in the on-policy learning, but also uses an advantage-based experience selection approach to transfer useful samples collected following the teacher policy in the off-policy learning.
  arXiv  Detail & Related papers  (2020-11-24T01:18:32Z)
• Measuring and Harnessing Transference in Multi-Task Learning [58.48659733262734]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We analyze the dynamics of information transfer, or transference, across tasks throughout training.
  arXiv  Detail & Related papers  (2020-10-29T08:25:43Z)
• Point Adversarial Self Mining: A Simple Method for Facial Expression Recognition [79.75964372862279]
  We propose Point Adversarial Self Mining (PASM) to improve the recognition accuracy in facial expression recognition. PASM uses a point adversarial attack method and a trained teacher network to locate the most informative position related to the target task. The adaptive learning materials generation and teacher/student update can be conducted more than one time, improving the network capability iteratively.
  arXiv  Detail & Related papers  (2020-08-26T06:39:24Z)
• Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach [55.83558520598304]
  We propose a brand new solution to reuse experiences and transfer value functions among multiple students via model distillation. We also describe how to design an efficient communication protocol to exploit heterogeneous knowledge. Our proposed framework, namely Learning and Teaching Categorical Reinforcement, shows promising performance on stabilizing and accelerating learning progress.
  arXiv  Detail & Related papers  (2020-02-06T11:31:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.