Related papers: Auxiliary task discovery through generate-and-test

Auxiliary task discovery through generate-and-test

URL: http://arxiv.org/abs/2210.14361v2
Date: Sat, 20 Jul 2024 16:54:39 GMT
Title: Auxiliary task discovery through generate-and-test
Authors: Banafsheh Rafiee, Sina Ghiassian, Jun Jin, Richard Sutton, Jun Luo, Adam White,
Abstract summary: Auxiliary tasks improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives. In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. We introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task.
Score: 7.800263769988046
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. Auxiliary tasks tend to improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives in addition to the main task of maximizing reward, and thus producing better representations. Typically these tasks are designed by people. Meta-learning offers a promising avenue for automatic task discovery; however, these methods are computationally expensive and challenging to tune in practice. In this paper, we explore a complementary approach to the auxiliary task discovery: continually generating new auxiliary tasks and preserving only those with high utility. We also introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task. Our discovery algorithm significantly outperforms random tasks and learning without auxiliary tasks across a suite of environments.

Related papers

Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks [33.98624423578388]
Auxiliary tasks improve representations learned by deep reinforcement learning agents. We derive a new family of auxiliary tasks based on the successor measure. We show that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms.
arXiv Detail & Related papers (2023-04-25T04:25:08Z)
Exploiting Contextual Structure to Generate Useful Auxiliary Tasks [12.236546713167945]
Reinforcement learning requires interaction with an environment, which is expensive for robots. We propose an approach that maximizes experience reuse while learning to solve a given task by generating and simultaneously learning useful auxiliary tasks.
arXiv Detail & Related papers (2023-03-09T05:11:30Z)
Auxiliary Learning as an Asymmetric Bargaining Game [50.826710465264505]
We propose a novel approach, named AuxiNash, for balancing tasks in auxiliary learning. We describe an efficient procedure for learning the bargaining power of tasks based on their contribution to the performance of the main task. We evaluate AuxiNash on multiple multi-task benchmarks and find that it consistently outperforms competing methods.
arXiv Detail & Related papers (2023-01-31T09:41:39Z)
Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
arXiv Detail & Related papers (2022-10-13T17:59:16Z)
AANG: Automating Auxiliary Learning [110.36191309793135]
We present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.
arXiv Detail & Related papers (2022-05-27T16:32:28Z)
Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors [52.37504333689262]
Using noisy labels in single-task learning increases the risk of over-fitting. Auxiliary tasks could improve the performance of the primary task learning during the same training.
arXiv Detail & Related papers (2021-12-02T08:40:42Z)
Learning Task Agnostic Skills with Data-driven Guidance [0.0]
This paper proposes a framework for guiding the skill discovery towards the subset of expert-visited states. We apply our method in various reinforcement learning tasks and show that such a projection results in more useful behaviours.
arXiv Detail & Related papers (2021-08-04T06:53:10Z)
Auxiliary Task Reweighting for Minimum-data Learning [118.69683270159108]
Supervised learning requires a large amount of training data, limiting its application where labeled data is scarce. To compensate for data scarcity, one possible method is to utilize auxiliary tasks to provide additional supervision for the main task. We propose a method to automatically reweight auxiliary tasks in order to reduce the data requirement on the main task.
arXiv Detail & Related papers (2020-10-16T08:45:37Z)
Auxiliary Learning by Implicit Differentiation [54.92146615836611]
Training neural networks with auxiliary tasks is a common practice for improving the performance on a main task of interest. Here, we propose a novel framework, AuxiLearn, that targets both challenges based on implicit differentiation. First, when useful auxiliaries are known, we propose learning a network that combines all losses into a single coherent objective function. Second, when no useful auxiliary task is known, we describe how to learn a network that generates a meaningful, novel auxiliary task.
arXiv Detail & Related papers (2020-06-22T19:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.