RIZE: Regularized Imitation Learning via Distributional Reinforcement Learning

• URL: http://arxiv.org/abs/2502.20089v1
• Date: Thu, 27 Feb 2025 13:47:29 GMT
• Title: RIZE: Regularized Imitation Learning via Distributional Reinforcement Learning
• Authors: Adib Karimi, Mohammad Mehdi Ebadzadeh,
• Abstract summary: We introduce a novel Inverse Reinforcement Learning (IRL) approach that overcomes limitations of fixed reward assignments.<n>We extend the Maximum Entropy IRL framework with a squared temporal-difference (TD) regularizer and adaptive targets, dynamically adjusted during training.<n>Our approach achieves state-of-the-art performance on challenging MuJoCo tasks, demonstrating expert-level results on the Humanoid task with only 3 demonstrations.
• Score: 0.3222802562733786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel Inverse Reinforcement Learning (IRL) approach that overcomes limitations of fixed reward assignments and constrained flexibility in implicit reward regularization. By extending the Maximum Entropy IRL framework with a squared temporal-difference (TD) regularizer and adaptive targets, dynamically adjusted during training, our method indirectly optimizes a reward function while incorporating reinforcement learning principles. Furthermore, we integrate distributional RL to capture richer return information. Our approach achieves state-of-the-art performance on challenging MuJoCo tasks, demonstrating expert-level results on the Humanoid task with only 3 demonstrations. Extensive experiments and ablation studies validate the effectiveness of our method, providing insights into adaptive targets and reward dynamics in imitation learning.

Related papers

• Reward-free World Models for Online Imitation Learning [25.304836126280424]
  We propose a novel approach to online imitation learning that leverages reward-free world models. Our method learns environmental dynamics entirely in latent spaces without reconstruction, enabling efficient and accurate modeling. We evaluate our method on a diverse set of benchmarks, including DMControl, MyoSuite, and ManiSkill2, demonstrating superior empirical performance compared to existing approaches.
  arXiv  Detail & Related papers  (2024-10-17T23:13:32Z)
• Learning Causally Invariant Reward Functions from Diverse Demonstrations [6.351909403078771]
  Inverse reinforcement learning methods aim to retrieve the reward function of a Markov decision process based on a dataset of expert demonstrations. This adaptation often exhibits overfitting to the expert data set when a policy is trained on the obtained reward function under distribution shift of the environment dynamics. In this work, we explore a novel regularization approach for inverse reinforcement learning methods based on the causal invariance principle with the goal of improved reward function generalization.
  arXiv  Detail & Related papers  (2024-09-12T12:56:24Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently.<n>Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Weighted Maximum Entropy Inverse Reinforcement Learning [22.269565708490468]
  We study inverse reinforcement learning (IRL) and imitation learning (IM) We propose a new way to improve the learning process by adding the maximum weight function to the entropy framework. Our framework and algorithms allow to learn both a reward (or policy) function and the structure of the entropy terms added to the Markov Decision Processes.
  arXiv  Detail & Related papers  (2022-08-20T06:02:07Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain [11.075036222901417]
  We propose an approach for inverse reinforcement learning from hetero-domain which learns a reward function in the simulator, drawing on the demonstrations from the real world. The intuition behind the method is that the reward function should not only be oriented to imitate the experts, but should encourage actions adjusted for the dynamics difference between the simulator and the real world.
  arXiv  Detail & Related papers  (2021-10-21T19:23:15Z)
• Off-Policy Imitation Learning from Observations [78.30794935265425]
  Learning from Observations (LfO) is a practical reinforcement learning scenario from which many applications can benefit. We propose a sample-efficient LfO approach that enables off-policy optimization in a principled manner. Our approach is comparable with state-of-the-art locomotion in terms of both sample-efficiency and performance.
  arXiv  Detail & Related papers  (2021-02-25T21:33:47Z)
• Regularized Inverse Reinforcement Learning [49.78352058771138]
  Inverse Reinforcement Learning (IRL) aims to facilitate a learner's ability to imitate expert behavior. Regularized IRL applies strongly convex regularizers to the learner's policy. We propose tractable solutions, and practical methods to obtain them, for regularized IRL.
  arXiv  Detail & Related papers  (2020-10-07T23:38:47Z)
• Reinforcement Learning through Active Inference [62.997667081978825]
  We show how ideas from active inference can augment traditional reinforcement learning approaches. We develop and implement a novel objective for decision making, which we term the free energy of the expected future. We demonstrate that the resulting algorithm successfully exploration and exploitation, simultaneously achieving robust performance on several challenging RL benchmarks with sparse, well-shaped, and no rewards.
  arXiv  Detail & Related papers  (2020-02-28T10:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.