Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing

• URL: http://arxiv.org/abs/2312.14472v2
• Date: Thu, 25 Jan 2024 14:35:05 GMT
• Title: Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing
• Authors: Jinmin He, Kai Li, Yifan Zang, Haobo Fu, Qiang Fu, Junliang Xing, Jian Cheng
• Abstract summary: Multi-task reinforcement learning endeavors to accomplish a set of different tasks with a single policy. This work presents a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of certain intermediate modules, thereby flexibly choosing different numbers of modules for each task. In addition, we design an automatic route-balancing mechanism to encourage continued routing exploration for unmastered tasks without disturbing the routing of mastered ones.
• Score: 26.44273671379482
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task reinforcement learning endeavors to accomplish a set of different tasks with a single policy. To enhance data efficiency by sharing parameters across multiple tasks, a common practice segments the network into distinct modules and trains a routing network to recombine these modules into task-specific policies. However, existing routing approaches employ a fixed number of modules for all tasks, neglecting that tasks with varying difficulties commonly require varying amounts of knowledge. This work presents a Dynamic Depth Routing (D2R) framework, which learns strategic skipping of certain intermediate modules, thereby flexibly choosing different numbers of modules for each task. Under this framework, we further introduce a ResRouting method to address the issue of disparate routing paths between behavior and target policies during off-policy training. In addition, we design an automatic route-balancing mechanism to encourage continued routing exploration for unmastered tasks without disturbing the routing of mastered ones. We conduct extensive experiments on various robotics manipulation tasks in the Meta-World benchmark, where D2R achieves state-of-the-art performance with significantly improved learning efficiency.

Related papers

• Contrastive Modules with Temporal Attention for Multi-Task Reinforcement Learning [29.14234496784581]
  We propose Contrastive Modules with Temporal Attention(CMTA) method for multi-task reinforcement learning. CMTA constrains the modules to be different from each other by contrastive learning and combining shared modules at a finer granularity than the task level. Experimental results show that CMTA outperforms learning each task individually for the first time and achieves substantial performance improvements.
  arXiv  Detail & Related papers  (2023-11-02T08:41:00Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Controllable Dynamic Multi-Task Architectures [92.74372912009127]
  We propose a controllable multi-task network that dynamically adjusts its architecture and weights to match the desired task preference as well as the resource constraints. We propose a disentangled training of two hypernetworks, by exploiting task affinity and a novel branching regularized loss, to take input preferences and accordingly predict tree-structured models with adapted weights.
  arXiv  Detail & Related papers  (2022-03-28T17:56:40Z)
• Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division [60.232542918414985]
  Multi-task learning often suffers from negative transfer, sharing information that should be task-specific. This is done by using proto-policies as modules to divide the tasks into simple sub-behaviours that can be shared. We also demonstrate its ability to autonomously divide the tasks into both shared and task-specific sub-behaviours.
  arXiv  Detail & Related papers  (2022-03-28T15:53:17Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)
• Multi-Task Reinforcement Learning with Soft Modularization [25.724764855681137]
  Multi-task learning is a very challenging problem in reinforcement learning. We introduce an explicit modularization technique on policy representation to alleviate this optimization issue. We show our method improves both sample efficiency and performance over strong baselines by a large margin.
  arXiv  Detail & Related papers  (2020-03-30T17:47:04Z)
• Dynamic Task Weighting Methods for Multi-task Networks in Autonomous Driving Systems [10.625400639764734]
  Deep multi-task networks are of particular interest for autonomous driving systems. We propose a novel method combining evolutionary meta-learning and task-based selective backpropagation. Our method outperforms state-of-the-art methods by a significant margin on a two-task application.
  arXiv  Detail & Related papers  (2020-01-07T18:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.