Multi-task Hierarchical Adversarial Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2305.12633v2
• Date: Wed, 28 Jun 2023 14:32:42 GMT
• Title: Multi-task Hierarchical Adversarial Inverse Reinforcement Learning
• Authors: Jiayu Chen, Dipesh Tamboli, Tian Lan, Vaneet Aggarwal
• Abstract summary: Multi-task Imitation Learning (MIL) aims to train a policy capable of performing a distribution of tasks based on multi-task expert demonstrations. Existing MIL algorithms suffer from low data efficiency and poor performance on complex long-horizontal tasks. We develop Multi-task Hierarchical Adversarial Inverse Reinforcement Learning (MH-AIRL) to learn hierarchically-structured multi-task policies.
• Score: 40.60364143826424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task Imitation Learning (MIL) aims to train a policy capable of performing a distribution of tasks based on multi-task expert demonstrations, which is essential for general-purpose robots. Existing MIL algorithms suffer from low data efficiency and poor performance on complex long-horizontal tasks. We develop Multi-task Hierarchical Adversarial Inverse Reinforcement Learning (MH-AIRL) to learn hierarchically-structured multi-task policies, which is more beneficial for compositional tasks with long horizons and has higher expert data efficiency through identifying and transferring reusable basic skills across tasks. To realize this, MH-AIRL effectively synthesizes context-based multi-task learning, AIRL (an IL approach), and hierarchical policy learning. Further, MH-AIRL can be adopted to demonstrations without the task or skill annotations (i.e., state-action pairs only) which are more accessible in practice. Theoretical justifications are provided for each module of MH-AIRL, and evaluations on challenging multi-task settings demonstrate superior performance and transferability of the multi-task policies learned with MH-AIRL as compared to SOTA MIL baselines.

Related papers

• Guiding Multi-agent Multi-task Reinforcement Learning by a Hierarchical Framework with Logical Reward Shaping [16.5526277899717]
  This study aims to design a multi-agent cooperative algorithm with logic reward shaping. Experiments have been conducted on various types of tasks in the Minecraft-like environment.
  arXiv  Detail & Related papers  (2024-11-02T09:03:23Z)
• MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning [74.43869839954168]
  We propose MTL-LoRA, which retains the advantages of low-rank adaptation while significantly enhancing multi-task learning capabilities. MTL-LoRA augments LoRA by incorporating additional task-adaptive parameters that differentiate task-specific information. This approach enables large language models (LLMs) pre-trained on general corpus to adapt to different target task domains with a limited number of trainable parameters.
  arXiv  Detail & Related papers  (2024-10-12T08:32:26Z)
• Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning [7.6201940008534175]
  HyGen is a novel hybrid MARL framework, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks.
  arXiv  Detail & Related papers  (2024-08-24T12:37:03Z)
• Variational Offline Multi-agent Skill Discovery [43.869625428099425]
  We propose two novel auto-encoder schemes to simultaneously capture subgroup- and temporal-level abstractions and form multi-agent skills. Our method can be applied to offline multi-task data, and the discovered subgroup skills can be transferred across relevant tasks without retraining.
  arXiv  Detail & Related papers  (2024-05-26T00:24:46Z)
• Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks [53.44714413181162]
  This paper shows that when an agent is trained on a sufficiently diverse set of tasks, a generic policy-sharing algorithm with myopic exploration design can be sample-efficient. To the best of our knowledge, this is the first theoretical demonstration of the "exploration benefits" of MTRL.
  arXiv  Detail & Related papers  (2024-03-03T22:57:44Z)
• LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning [122.47938710284784]
  We propose a novel framework for learning dynamic subtask assignment (LDSA) in cooperative MARL. To reasonably assign agents to different subtasks, we propose an ability-based subtask selection strategy. We show that LDSA learns reasonable and effective subtask assignment for better collaboration.
  arXiv  Detail & Related papers  (2022-05-05T10:46:16Z)
• Modular Adaptive Policy Selection for Multi-Task Imitation Learning through Task Division [60.232542918414985]
  Multi-task learning often suffers from negative transfer, sharing information that should be task-specific. This is done by using proto-policies as modules to divide the tasks into simple sub-behaviours that can be shared. We also demonstrate its ability to autonomously divide the tasks into both shared and task-specific sub-behaviours.
  arXiv  Detail & Related papers  (2022-03-28T15:53:17Z)
• Gradient Surgery for Multi-Task Learning [119.675492088251]
  Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient.
  arXiv  Detail & Related papers  (2020-01-19T06:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.