Exploiting Contextual Structure to Generate Useful Auxiliary Tasks

• URL: http://arxiv.org/abs/2303.05038v2
• Date: Thu, 4 Apr 2024 05:37:52 GMT
• Title: Exploiting Contextual Structure to Generate Useful Auxiliary Tasks
• Authors: Benedict Quartey, Ankit Shah, George Konidaris,
• Abstract summary: Reinforcement learning requires interaction with an environment, which is expensive for robots. We propose an approach that maximizes experience reuse while learning to solve a given task by generating and simultaneously learning useful auxiliary tasks.
• Score: 12.236546713167945
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning requires interaction with an environment, which is expensive for robots. This constraint necessitates approaches that work with limited environmental interaction by maximizing the reuse of previous experiences. We propose an approach that maximizes experience reuse while learning to solve a given task by generating and simultaneously learning useful auxiliary tasks. To generate these tasks, we construct an abstract temporal logic representation of the given task and leverage large language models to generate context-aware object embeddings that facilitate object replacements. Counterfactual reasoning and off-policy methods allow us to simultaneously learn these auxiliary tasks while solving the given target task. We combine these insights into a novel framework for multitask reinforcement learning and experimentally show that our generated auxiliary tasks share similar underlying exploration requirements as the given task, thereby maximizing the utility of directed exploration. Our approach allows agents to automatically learn additional useful policies without extra environment interaction.

Related papers

• Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks [33.98624423578388]
  Auxiliary tasks improve representations learned by deep reinforcement learning agents. We derive a new family of auxiliary tasks based on the successor measure. We show that proto-value networks produce rich features that may be used to obtain performance comparable to established algorithms.
  arXiv  Detail & Related papers  (2023-04-25T04:25:08Z)
• Auxiliary task discovery through generate-and-test [7.800263769988046]
  Auxiliary tasks improve data efficiency by forcing the agent to learn auxiliary prediction and control objectives. In this paper, we explore an approach to auxiliary task discovery in reinforcement learning based on ideas from representation learning. We introduce a new measure of auxiliary tasks' usefulness based on how useful the features induced by them are for the main task.
  arXiv  Detail & Related papers  (2022-10-25T22:04:37Z)
• Saliency-Regularized Deep Multi-Task Learning [7.3810864598379755]
  Multitask learning enforces multiple learning tasks to share knowledge to improve their generalization abilities. Modern deep multitask learning can jointly learn latent features and task sharing, but they are obscure in task relation. This paper proposes a new multitask learning framework that jointly learns latent features and explicit task relations.
  arXiv  Detail & Related papers  (2022-07-03T20:26:44Z)
• AANG: Automating Auxiliary Learning [110.36191309793135]
  We present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.
  arXiv  Detail & Related papers  (2022-05-27T16:32:28Z)
• Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery [22.32327908453603]
  Current reinforcement learning (RL) in robotics often experiences difficulty in generalizing to new downstream tasks. We propose a framework that pre-trains the agent in a task-agnostic manner without access to the task-specific reward. We show that our approach achieves the most diverse interacting behavior and significantly improves sample efficiency in downstream tasks.
  arXiv  Detail & Related papers  (2022-04-29T06:57:46Z)
• Skill-based Meta-Reinforcement Learning [65.31995608339962]
  We devise a method that enables meta-learning on long-horizon, sparse-reward tasks. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning.
  arXiv  Detail & Related papers  (2022-04-25T17:58:19Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Multi-Task Reinforcement Learning with Context-based Representations [43.93866702838777]
  We propose an efficient approach to knowledge transfer through the use of multiple context-dependent, composable representations across a family of tasks. We use the proposed approach to obtain state-of-the-art results in Meta-World, a challenging multi-task benchmark consisting of 50 distinct robotic manipulation tasks.
  arXiv  Detail & Related papers  (2021-02-11T18:41:27Z)
• Latent Skill Planning for Exploration and Transfer [49.25525932162891]
  In this paper, we investigate how these two approaches can be integrated into a single reinforcement learning agent. We leverage the idea of partial amortization for fast adaptation at test time. We demonstrate the benefits of our design decisions across a suite of challenging locomotion tasks.
  arXiv  Detail & Related papers  (2020-11-27T18:40:03Z)
• Adaptive Procedural Task Generation for Hard-Exploration Problems [78.20918366839399]
  We introduce Adaptive Procedural Task Generation (APT-Gen) to facilitate reinforcement learning in hard-exploration problems. At the heart of our approach is a task generator that learns to create tasks from a parameterized task space via a black-box procedural generation module. To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks.
  arXiv  Detail & Related papers  (2020-07-01T09:38:51Z)
• Weakly-Supervised Reinforcement Learning for Controllable Behavior [126.04932929741538]
  Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks. In many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve. We introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks.
  arXiv  Detail & Related papers  (2020-04-06T17:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.