Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2408.15593v1
• Date: Wed, 28 Aug 2024 07:36:20 GMT
• Title: Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
• Authors: Minjong Yoo, Sangwoo Cho, Honguk Woo,
• Abstract summary: Reinforcement learning (RL) with diverse offline datasets can have the advantage of leveraging the relation of multiple tasks. We present a skill-based multi-task RL technique on heterogeneous datasets that are generated by behavior policies of different quality. We show that our multi-task offline RL approach is robust to the mixed configurations of different-quality datasets.
• Score: 11.790581500542439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) with diverse offline datasets can have the advantage of leveraging the relation of multiple tasks and the common skills learned across those tasks, hence allowing us to deal with real-world complex problems efficiently in a data-driven way. In offline RL where only offline data is used and online interaction with the environment is restricted, it is yet difficult to achieve the optimal policy for multiple tasks, especially when the data quality varies for the tasks. In this paper, we present a skill-based multi-task RL technique on heterogeneous datasets that are generated by behavior policies of different quality. To learn the shareable knowledge across those datasets effectively, we employ a task decomposition method for which common skills are jointly learned and used as guidance to reformulate a task in shared and achievable subtasks. In this joint learning, we use Wasserstein auto-encoder (WAE) to represent both skills and tasks on the same latent space and use the quality-weighted loss as a regularization term to induce tasks to be decomposed into subtasks that are more consistent with high-quality skills than others. To improve the performance of offline RL agents learned on the latent space, we also augment datasets with imaginary trajectories relevant to high-quality skills for each task. Through experiments, we show that our multi-task offline RL approach is robust to the mixed configurations of different-quality datasets and it outperforms other state-of-the-art algorithms for several robotic manipulation tasks and drone navigation tasks.

Related papers

• Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning [116.87367592920171]
  Offline Reinforcement Learning (RL) has shown promising results in learning a task-specific policy from a fixed dataset. In scenarios where the dataset for a specific task is limited, a natural approach is to improve offline RL with datasets from other tasks. We propose an uncertainty-based Multi-Task Data Sharing (MTDS) approach that shares the entire dataset without data selection.
  arXiv  Detail & Related papers  (2024-04-30T08:16:52Z)
• Robust Policy Learning via Offline Skill Diffusion [6.876580618014666]
  We present a novel offline skill learning framework, DuSkill. DuSkill employs a guided Diffusion model to generate versatile skills extended from the limited skills in datasets. We show that DuSkill outperforms other skill-based imitation learning and RL algorithms for several long-horizon tasks.
  arXiv  Detail & Related papers  (2024-03-01T02:00:44Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Generalizable Task Representation Learning for Offline Meta-Reinforcement Learning with Data Limitations [22.23114883485924]
  We propose a novel algorithm called GENTLE for learning generalizable task representations in the face of data limitations. GENTLE employs Task Auto-Encoder(TAE), which is an encoder-decoder architecture to extract the characteristics of the tasks. To alleviate the effect of limited behavior diversity, we construct pseudo-transitions to align the data distribution used to train TAE with the data distribution encountered during testing.
  arXiv  Detail & Related papers  (2023-12-26T07:02:12Z)
• Efficient Computation Sharing for Multi-Task Visual Scene Understanding [16.727967046330125]
  Multi-task learning can conserve resources by sharing knowledge across different tasks. We present a novel- and parameter-sharing framework that balances efficiency and accuracy to perform multiple visual tasks.
  arXiv  Detail & Related papers  (2023-03-16T21:47:40Z)
• Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks [37.73239471412444]
  We introduce Active Task Randomization (ATR), an approach that learns robust skills through the unsupervised generation of training tasks. ATR selects suitable tasks, which consist of an initial environment state and manipulation goal, for learning robust skills by balancing the diversity and feasibility of the tasks. We demonstrate that the learned skills can be composed by a task planner to solve unseen sequential manipulation problems based on visual inputs.
  arXiv  Detail & Related papers  (2022-11-11T11:24:55Z)
• Skill-based Meta-Reinforcement Learning [65.31995608339962]
  We devise a method that enables meta-learning on long-horizon, sparse-reward tasks. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning.
  arXiv  Detail & Related papers  (2022-04-25T17:58:19Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning [119.85598717477016]
  We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks. We develop a simple technique for data-sharing in multi-task offline RL that routes data based on the improvement over the task-specific data.
  arXiv  Detail & Related papers  (2021-09-16T17:34:06Z)
• Gradient Surgery for Multi-Task Learning [119.675492088251]
  Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient.
  arXiv  Detail & Related papers  (2020-01-19T06:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.