TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL

• URL: http://arxiv.org/abs/2002.06659v2
• Date: Mon, 8 Mar 2021 17:14:57 GMT
• Title: TempLe: Learning Template of Transitions for Sample Efficient Multi-task RL
• Authors: Yanchao Sun, Xiangyu Yin, Furong Huang
• Abstract summary: TempLe is the first PAC-MDP method for multi-task reinforcement learning. We present two algorithms for an "online" and a "finite-model" setting respectively. We prove that our proposed TempLe algorithms achieve much lower sample complexity than single-task learners or state-of-the-art multi-task methods.
• Score: 18.242904106537654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transferring knowledge among various environments is important to efficiently learn multiple tasks online. Most existing methods directly use the previously learned models or previously learned optimal policies to learn new tasks. However, these methods may be inefficient when the underlying models or optimal policies are substantially different across tasks. In this paper, we propose Template Learning (TempLe), the first PAC-MDP method for multi-task reinforcement learning that could be applied to tasks with varying state/action space. TempLe generates transition dynamics templates, abstractions of the transition dynamics across tasks, to gain sample efficiency by extracting similarities between tasks even when their underlying models or optimal policies have limited commonalities. We present two algorithms for an "online" and a "finite-model" setting respectively. We prove that our proposed TempLe algorithms achieve much lower sample complexity than single-task learners or state-of-the-art multi-task methods. We show via systematically designed experiments that our TempLe method universally outperforms the state-of-the-art multi-task methods (PAC-MDP or not) in various settings and regimes.

Related papers

• On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion [23.63688816017186]
  Existing weak-to-strong methods often employ a static knowledge transfer ratio and a single small model for transferring complex knowledge. We propose a dynamic logit fusion approach that works with a series of task-specific small models, each specialized in a different task. Our method closes the performance gap by 96.4% in single-task scenarios and by 86.3% in multi-task scenarios.
  arXiv  Detail & Related papers  (2024-06-17T03:07:41Z)
• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning [101.66860222415512]
  Multi-Task Diffusion Model (textscMTDiff) is a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis. For generative planning, we find textscMTDiff outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D.
  arXiv  Detail & Related papers  (2023-05-29T05:20:38Z)
• QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing [18.127823952220123]
  Multi-task reinforcement learning (MTRL) aims to learn several tasks simultaneously for better sample efficiency than learning them separately. We introduce a new framework for sharing behavioral policies across tasks, which can be used in addition to existing MTRL methods.
  arXiv  Detail & Related papers  (2023-02-01T18:58:20Z)
• Multi-task Active Learning for Pre-trained Transformer-based Models [22.228551277598804]
  Multi-task learning, in which several tasks are jointly learned by a single model, allows NLP models to share information from multiple annotations. This technique requires annotating the same text with multiple annotation schemes which may be costly and laborious. Active learning (AL) has been demonstrated to optimize annotation processes by iteratively selecting unlabeled examples.
  arXiv  Detail & Related papers  (2022-08-10T14:54:13Z)
• Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports [2.5953185061765884]
  We show that a single multi-tasking model can match the performance of task specific models. We validate our observations on our internal radiologist-annotated datasets on the cervical and lumbar spine.
  arXiv  Detail & Related papers  (2022-05-06T01:51:19Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• The Effect of Diversity in Meta-Learning [79.56118674435844]
  Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the model's performance. We study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms.
  arXiv  Detail & Related papers  (2022-01-27T19:39:07Z)
• UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning [64.638804236566]
  We propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup. Remarkably, on the GLUE benchmark, UniPELT consistently achieves 13pt gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups.
  arXiv  Detail & Related papers  (2021-10-14T17:40:08Z)
• Controllable Pareto Multi-Task Learning [55.945680594691076]
  A multi-task learning system aims at solving multiple related tasks at the same time. With a fixed model capacity, the tasks would be conflicted with each other, and the system usually has to make a trade-off among learning all of them together. This work proposes a novel controllable multi-task learning framework, to enable the system to make real-time trade-off control among different tasks with a single model.
  arXiv  Detail & Related papers  (2020-10-13T11:53:55Z)
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling [126.69933134648541]
  We present a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data.
  arXiv  Detail & Related papers  (2020-06-12T13:34:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.