Quality Diversity Imitation Learning

• URL: http://arxiv.org/abs/2410.06151v1
• Date: Tue, 8 Oct 2024 15:49:33 GMT
• Title: Quality Diversity Imitation Learning
• Authors: Zhenglin Wan, Xingrui Yu, David Mark Bossens, Yueming Lyu, Qing Guo, Flint Xiaofeng Fan, Ivor Tsang,
• Abstract summary: We introduce the first generic framework for Quality Diversity Imitation Learning (QD-IL) Our framework integrates the principles of quality diversity with adversarial imitation learning (AIL) methods, and can potentially improve any inverse reinforcement learning (IRL) method. Our method even achieves 2x expert performance in the most challenging Humanoid environment.
• Score: 9.627530753815968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Imitation learning (IL) has shown great potential in various applications, such as robot control. However, traditional IL methods are usually designed to learn only one specific type of behavior since demonstrations typically correspond to a single expert. In this work, we introduce the first generic framework for Quality Diversity Imitation Learning (QD-IL), which enables the agent to learn a broad range of skills from limited demonstrations. Our framework integrates the principles of quality diversity with adversarial imitation learning (AIL) methods, and can potentially improve any inverse reinforcement learning (IRL) method. Empirically, our framework significantly improves the QD performance of GAIL and VAIL on the challenging continuous control tasks derived from Mujoco environments. Moreover, our method even achieves 2x expert performance in the most challenging Humanoid environment.

Related papers

• Imitation from Diverse Behaviors: Wasserstein Quality Diversity Imitation Learning with Single-Step Archive Exploration [37.836675202590406]
  This work introduces Wasserstein Quality Diversity Imitation Learning (WQDIL) It improves the stability of imitation learning in the quality diversity setting with latent adversarial training based on a Wasserstein Auto-Encoder (WAE) It mitigates a behavior-overfitting issue using a measure-conditioned reward function with a single-step archive exploration bonus.
  arXiv  Detail & Related papers  (2024-11-11T13:11:18Z)
• Latent-Predictive Empowerment: Measuring Empowerment without a Simulator [56.53777237504011]
  We present Latent-Predictive Empowerment (LPE), an algorithm that can compute empowerment in a more practical manner. LPE learns large skillsets by maximizing an objective that is a principled replacement for the mutual information between skills and states.
  arXiv  Detail & Related papers  (2024-10-15T00:41:18Z)
• Explorative Imitation Learning: A Path Signature Approach for Continuous Environments [9.416194245966022]
  Continuous Imitation Learning from Observation (CILO) is a new method augmenting imitation learning with two important features. CILO exploration allows for more diverse state transitions, requiring less expert trajectories and resulting in fewer training iterations. It has the best overall performance of all imitation learning methods in all environments, outperforming the expert in two of them.
  arXiv  Detail & Related papers  (2024-07-05T20:25:39Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
  Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
  arXiv  Detail & Related papers  (2024-03-11T17:49:18Z)
• Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL) [1.0057319866872687]
  This paper presents a method called Continuous Mean-Zero Disagreement-Regularized Imitation Learning (CMZ-DRIL) CMZ-DRIL uses reinforcement learning to minimize uncertainty among an ensemble of agents trained to model the expert demonstrations. As demonstrated in a waypoint-navigation environment and in two MuJoCo environments, CMZ-DRIL can generate performant agents that behave more similarly to the expert.
  arXiv  Detail & Related papers  (2024-03-02T01:40:37Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations [60.241144377865716]
  We consider the imitation of sub-optimal demonstrations, with both a small clean demonstration set and a large noisy set. We propose method by evaluating and imitating at the sub-demonstration level, encoding action primitives of varying quality into different skills.
  arXiv  Detail & Related papers  (2023-06-13T17:24:37Z)
• Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions [19.626042478612572]
  We propose a cooperative adversarial method for obtaining versatile policies with controllable skill sets from unlabeled datasets. We show that by utilizing unsupervised skill discovery in the generative imitation learning framework, novel and useful skills emerge with successful task fulfillment. Finally, the obtained versatile policies are tested on an agile quadruped robot called Solo 8 and present faithful replications of diverse skills encoded in the demonstrations.
  arXiv  Detail & Related papers  (2022-09-16T12:49:04Z)
• Soft Expert Reward Learning for Vision-and-Language Navigation [94.86954695912125]
  Vision-and-Language Navigation (VLN) requires an agent to find a specified spot in an unseen environment by following natural language instructions. We introduce a Soft Expert Reward Learning (SERL) model to overcome the reward engineering designing and generalisation problems of the VLN task.
  arXiv  Detail & Related papers  (2020-07-21T14:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.