Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation

• URL: http://arxiv.org/abs/2411.16532v1
• Date: Mon, 25 Nov 2024 16:18:39 GMT
• Title: Continual Deep Reinforcement Learning with Task-Agnostic Policy Distillation
• Authors: Muhammad Burhan Hafez, Kerim Erekmen,
• Abstract summary: The Task-Agnostic Policy Distillation (TAPD) framework is introduced. This paper addresses the problem of continual learning. By utilizing task-agnostic distilled knowledge, the agent can solve downstream tasks more efficiently.
• Score: 0.0
• License:
• Abstract: Central to the development of universal learning systems is the ability to solve multiple tasks without retraining from scratch when new data arrives. This is crucial because each task requires significant training time. Addressing the problem of continual learning necessitates various methods due to the complexity of the problem space. This problem space includes: (1) addressing catastrophic forgetting to retain previously learned tasks, (2) demonstrating positive forward transfer for faster learning, (3) ensuring scalability across numerous tasks, and (4) facilitating learning without requiring task labels, even in the absence of clear task boundaries. In this paper, the Task-Agnostic Policy Distillation (TAPD) framework is introduced. This framework alleviates problems (1)-(4) by incorporating a task-agnostic phase, where an agent explores its environment without any external goal and maximizes only its intrinsic motivation. The knowledge gained during this phase is later distilled for further exploration. Therefore, the agent acts in a self-supervised manner by systematically seeking novel states. By utilizing task-agnostic distilled knowledge, the agent can solve downstream tasks more efficiently, leading to improved sample efficiency. Our code is available at the repository: https://github.com/wabbajack1/TAPD.

Related papers

• Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations [24.041217922654738]
  Continuous control problems can be formulated as sparse-reward reinforcement learning (RL) tasks. Online RL methods can automatically explore the state space to solve each new task. However, discovering sequences of actions that lead to a non-zero reward becomes exponentially more difficult as the task horizon increases. We introduce a systematic reward-shaping framework that distills the information contained in 1) a task-agnostic prior data set and 2) a small number of task-specific expert demonstrations.
  arXiv  Detail & Related papers  (2024-12-02T04:37:12Z)
• Multitask Learning with No Regret: from Improved Confidence Bounds to Active Learning [79.07658065326592]
  Quantifying uncertainty in the estimated tasks is of pivotal importance for many downstream applications, such as online or active learning. We provide novel multitask confidence intervals in the challenging setting when neither the similarity between tasks nor the tasks' features are available to the learner. We propose a novel online learning algorithm that achieves such improved regret without knowing this parameter in advance.
  arXiv  Detail & Related papers  (2023-08-03T13:08:09Z)
• A Unified Meta-Learning Framework for Dynamic Transfer Learning [42.34180707803632]
  We propose a generic meta-learning framework L2E for modeling the knowledge transferability on dynamic tasks. L2E enjoys the following properties: (1) effective knowledge transferability across dynamic tasks; (2) fast adaptation to the new target task; (3) mitigation of catastrophic forgetting on historical target tasks; and (4) flexibility in incorporating any existing static transfer learning algorithms.
  arXiv  Detail & Related papers  (2022-07-05T02:56:38Z)
• Generalizing to New Tasks via One-Shot Compositional Subgoals [23.15624959305799]
  The ability to generalize to previously unseen tasks with little to no supervision is a key challenge in modern machine learning research. We introduce CASE which attempts to address these issues by training an Imitation Learning agent using adaptive "near future" subgoals. Our experiments show that the proposed approach consistently outperforms the previous state-of-the-art compositional Imitation Learning approach by 30%.
  arXiv  Detail & Related papers  (2022-05-16T14:30:11Z)
• Skill-based Meta-Reinforcement Learning [65.31995608339962]
  We devise a method that enables meta-learning on long-horizon, sparse-reward tasks. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning.
  arXiv  Detail & Related papers  (2022-04-25T17:58:19Z)
• ConTinTin: Continual Learning from Task Instructions [101.36836925135091]
  This work defines a new learning paradigm ConTinTin, in which a system should learn a sequence of new tasks one by one, each task is explained by a piece of textual instruction. To our knowledge, this is the first time to study ConTinTin in NLP.
  arXiv  Detail & Related papers  (2022-03-16T10:27:18Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Planning to Explore via Self-Supervised World Models [120.31359262226758]
  Plan2Explore is a self-supervised reinforcement learning agent. We present a new approach to self-supervised exploration and fast adaptation to new tasks. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods.
  arXiv  Detail & Related papers  (2020-05-12T17:59:45Z)
• Weakly-Supervised Reinforcement Learning for Controllable Behavior [126.04932929741538]
  Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. In many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. We introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks.
  arXiv  Detail & Related papers  (2020-04-06T17:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.