Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning

• URL: http://arxiv.org/abs/2505.02228v1
• Date: Sun, 04 May 2025 19:32:48 GMT
• Title: Coupled Distributional Random Expert Distillation for World Model Online Imitation Learning
• Authors: Shangzhe Li, Zhiao Huang, Hao Su,
• Abstract summary: Imitation Learning (IL) has achieved remarkable success across various domains, including robotics, autonomous driving, and healthcare, by enabling agents to learn complex behaviors from expert demonstrations.<n>Existing IL methods often face instability challenges, particularly when relying on adversarial reward or value formulations in world model frameworks.<n>We propose a novel approach to online imitation learning that addresses these limitations through a reward model based on random network distillation (RND) for density estimation.
• Score: 25.304836126280424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation Learning (IL) has achieved remarkable success across various domains, including robotics, autonomous driving, and healthcare, by enabling agents to learn complex behaviors from expert demonstrations. However, existing IL methods often face instability challenges, particularly when relying on adversarial reward or value formulations in world model frameworks. In this work, we propose a novel approach to online imitation learning that addresses these limitations through a reward model based on random network distillation (RND) for density estimation. Our reward model is built on the joint estimation of expert and behavioral distributions within the latent space of the world model. We evaluate our method across diverse benchmarks, including DMControl, Meta-World, and ManiSkill2, showcasing its ability to deliver stable performance and achieve expert-level results in both locomotion and manipulation tasks. Our approach demonstrates improved stability over adversarial methods while maintaining expert-level performance.

Related papers

• A Model-Based Approach to Imitation Learning through Multi-Step Predictions [8.888213496593556]
  We present a novel model-based imitation learning framework inspired by model predictive control.<n>Our method outperforms traditional behavior cloning numerical benchmarks.<n>We provide theoretical guarantees on the sample complexity and error bounds of our method, offering insights into its convergence properties.
  arXiv  Detail & Related papers  (2025-04-18T02:19:30Z)
• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  This work advances model-based reinforcement learning by addressing the challenges of long-horizon prediction, error accumulation, and sim-to-real transfer.<n>By providing a scalable and robust framework, the introduced methods pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)
• Reward-free World Models for Online Imitation Learning [25.304836126280424]
  We propose a novel approach to online imitation learning that leverages reward-free world models.<n>Our method learns environmental dynamics entirely in latent spaces without reconstruction, enabling efficient and accurate modeling.<n>We evaluate our method on a diverse set of benchmarks, including DMControl, MyoSuite, and ManiSkill2, demonstrating superior empirical performance compared to existing approaches.
  arXiv  Detail & Related papers  (2024-10-17T23:13:32Z)
• Learning Multiple Probabilistic Decisions from Latent World Model in Autonomous Driving [40.4491758280365]
  Autoregressive world model exhibits robust generalization capabilities but encounters difficulties in deriving actions due to insufficient uncertainty modeling and self-delusion. We propose LatentDriver, a framework models the environment's next states and the ego vehicle's possible actions as a mixture distribution. LatentDriver surpasses state-of-the-art reinforcement learning and imitation learning methods, achieving expert-level performance.
  arXiv  Detail & Related papers  (2024-09-24T04:26:24Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Leveraging World Model Disentanglement in Value-Based Multi-Agent Reinforcement Learning [18.651307543537655]
  We propose a novel model-based multi-agent reinforcement learning approach named Value Decomposition Framework with Disentangled World Model. We present experimental results in Easy, Hard, and Super-Hard StarCraft II micro-management challenges to demonstrate that our method achieves high sample efficiency and exhibits superior performance in defeating the enemy armies compared to other baselines.
  arXiv  Detail & Related papers  (2023-09-08T22:12:43Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• Mimicking Better by Matching the Approximate Action Distribution [48.95048003354255]
  We introduce MAAD, a novel, sample-efficient on-policy algorithm for Imitation Learning from Observations. We show that it requires considerable fewer interactions to achieve expert performance, outperforming current state-of-the-art on-policy methods.
  arXiv  Detail & Related papers  (2023-06-16T12:43:47Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning [114.69155066932046]
  This work proposes a novel. Teacher with Decentralized Student (C TDS) framework, which consists of a teacher model and a student model. Specifically, the teacher model allocates the team reward by learning individual Q-values conditioned on global observation. The student model utilizes the partial observations to approximate the Q-values estimated by the teacher model.
  arXiv  Detail & Related papers  (2022-03-16T06:03:14Z)
• Soft Expert Reward Learning for Vision-and-Language Navigation [94.86954695912125]
  Vision-and-Language Navigation (VLN) requires an agent to find a specified spot in an unseen environment by following natural language instructions. We introduce a Soft Expert Reward Learning (SERL) model to overcome the reward engineering designing and generalisation problems of the VLN task.
  arXiv  Detail & Related papers  (2020-07-21T14:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.