Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- URL: http://arxiv.org/abs/2401.09561v1
- Date: Wed, 17 Jan 2024 19:31:21 GMT
- Title: Sharing Knowledge in Multi-Task Deep Reinforcement Learning
- Authors: Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, Jan
Peters
- Abstract summary: We study the benefit of sharing representations among tasks to enable the effective use of deep neural networks in Multi-Task Reinforcement Learning.
We prove this by providing theoretical guarantees that highlight the conditions for which is convenient to share representations among tasks.
- Score: 57.38874587065694
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We study the benefit of sharing representations among tasks to enable the
effective use of deep neural networks in Multi-Task Reinforcement Learning. We
leverage the assumption that learning from different tasks, sharing common
properties, is helpful to generalize the knowledge of them resulting in a more
effective feature extraction compared to learning a single task. Intuitively,
the resulting set of features offers performance benefits when used by
Reinforcement Learning algorithms. We prove this by providing theoretical
guarantees that highlight the conditions for which is convenient to share
representations among tasks, extending the well-known finite-time bounds of
Approximate Value-Iteration to the multi-task setting. In addition, we
complement our analysis by proposing multi-task extensions of three
Reinforcement Learning algorithms that we empirically evaluate on widely used
Reinforcement Learning benchmarks showing significant improvements over the
single-task counterparts in terms of sample efficiency and performance.
Related papers
- Learning Representation for Multitask learning through Self Supervised Auxiliary learning [3.236198583140341]
In the hard parameter sharing approach, an encoder shared through multiple tasks generates data representations passed to task-specific predictors.
We propose Dummy Gradient norm Regularization that aims to improve the universality of the representations generated by the shared encoder.
We show that DGR effectively improves the quality of the shared representations, leading to better multi-task prediction performances.
arXiv Detail & Related papers (2024-09-25T06:08:35Z) - Sample Efficient Myopic Exploration Through Multitask Reinforcement
Learning with Diverse Tasks [53.44714413181162]
This paper shows that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design can be sample-efficient.
To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL.
arXiv Detail & Related papers (2024-03-03T22:57:44Z) - Generalization Performance of Transfer Learning: Overparameterized and
Underparameterized Regimes [61.22448274621503]
In real-world applications, tasks often exhibit partial similarity, where certain aspects are similar while others are different or irrelevant.
Our study explores various types of transfer learning, encompassing two options for parameter transfer.
We provide practical guidelines for determining the number of features in the common and task-specific parts for improved generalization performance.
arXiv Detail & Related papers (2023-06-08T03:08:40Z) - Proto-Value Networks: Scaling Representation Learning with Auxiliary
Tasks [33.98624423578388]
Auxiliary tasks improve representations learned by deep reinforcement learning agents.
We derive a new family of auxiliary tasks based on the successor measure.
We show that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms.
arXiv Detail & Related papers (2023-04-25T04:25:08Z) - Auxiliary Learning as an Asymmetric Bargaining Game [50.826710465264505]
We propose a novel approach, named AuxiNash, for balancing tasks in auxiliary learning.
We describe an efficient procedure for learning the bargaining power of tasks based on their contribution to the performance of the main task.
We evaluate AuxiNash on multiple multi-task benchmarks and find that it consistently outperforms competing methods.
arXiv Detail & Related papers (2023-01-31T09:41:39Z) - Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task.
We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
arXiv Detail & Related papers (2022-10-13T17:59:16Z) - Saliency-Regularized Deep Multi-Task Learning [7.3810864598379755]
Multitask learning enforces multiple learning tasks to share knowledge to improve their generalization abilities.
Modern deep multitask learning can jointly learn latent features and task sharing, but they are obscure in task relation.
This paper proposes a new multitask learning framework that jointly learns latent features and explicit task relations.
arXiv Detail & Related papers (2022-07-03T20:26:44Z) - Leveraging convergence behavior to balance conflicting tasks in
multi-task learning [3.6212652499950138]
Multi-Task Learning uses correlated tasks to improve performance generalization.
Tasks often conflict with each other, which makes it challenging to define how the gradients of multiple tasks should be combined.
We propose a method that takes into account temporal behaviour of the gradients to create a dynamic bias that adjust the importance of each task during the backpropagation.
arXiv Detail & Related papers (2022-04-14T01:52:34Z) - Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene
Text Recognition [60.36540008537054]
In this work, we excavate the implicit task, character counting within the traditional text recognition, without additional labor annotation cost.
We design a two-branch reciprocal feature learning framework in order to adequately utilize the features from both the tasks.
Experiments on 7 benchmarks show the advantages of the proposed methods in both text recognition and the new-built character counting tasks.
arXiv Detail & Related papers (2021-05-13T12:27:35Z) - Efficient Reinforcement Learning in Resource Allocation Problems Through
Permutation Invariant Multi-task Learning [6.247939901619901]
We show that in certain settings, the available data can be dramatically increased through a form of multi-task learning.
We provide a theoretical performance bound for the gain in sample efficiency under this setting.
This motivates a new approach to multi-task learning, which involves the design of an appropriate neural network architecture and a prioritized task-sampling strategy.
arXiv Detail & Related papers (2021-02-18T14:13:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.