Inverse Reinforcement Learning in the Continuous Setting with Formal Guarantees

• URL: http://arxiv.org/abs/2102.07937v1
• Date: Tue, 16 Feb 2021 03:17:23 GMT
• Title: Inverse Reinforcement Learning in the Continuous Setting with Formal Guarantees
• Authors: Gregory Dexter, Kevin Bello, and Jean Honorio
• Abstract summary: Inverse Reinforcement Learning (IRL) is the problem of finding a reward function which describes observed/known expert behavior. We provide a new IRL algorithm for the continuous state space setting with unknown transition dynamics.
• Score: 31.122125783516726
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverse Reinforcement Learning (IRL) is the problem of finding a reward function which describes observed/known expert behavior. IRL is useful for automated control in situations where the reward function is difficult to specify manually, which impedes reinforcement learning. We provide a new IRL algorithm for the continuous state space setting with unknown transition dynamics by modeling the system using a basis of orthonormal functions. We provide a proof of correctness and formal guarantees on the sample and time complexity of our algorithm.

Related papers

• Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching [23.600285251963395]
  In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment. Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimize the reward through repeated RL procedures. We propose a novel approach to IRL by direct policy optimization, exploiting a linear factorization of the return as the inner product of successor features and a reward vector.
  arXiv  Detail & Related papers  (2024-11-11T14:05:50Z)
• Automated Feature Selection for Inverse Reinforcement Learning [7.278033100480175]
  Inverse reinforcement learning (IRL) is an imitation learning approach to learning reward functions from expert demonstrations. We propose a method that employs basis functions to form a candidate set of features. We demonstrate the approach's effectiveness by recovering reward functions that capture expert policies.
  arXiv  Detail & Related papers  (2024-03-22T10:05:21Z)
• Resilient Constrained Learning [94.27081585149836]
  This paper presents a constrained learning approach that adapts the requirements while simultaneously solving the learning task. We call this approach resilient constrained learning after the term used to describe ecological systems that adapt to disruptions by modifying their operation.
  arXiv  Detail & Related papers  (2023-06-04T18:14:18Z)
• Weighted Maximum Entropy Inverse Reinforcement Learning [22.269565708490468]
  We study inverse reinforcement learning (IRL) and imitation learning (IM) We propose a new way to improve the learning process by adding the maximum weight function to the entropy framework. Our framework and algorithms allow to learn both a reward (or policy) function and the structure of the entropy terms added to the Markov Decision Processes.
  arXiv  Detail & Related papers  (2022-08-20T06:02:07Z)
• Stabilizing Q-learning with Linear Architectures for Provably Efficient Learning [53.17258888552998]
  This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation. We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
  arXiv  Detail & Related papers  (2022-06-01T23:26:51Z)
• Temporal Abstractions-Augmented Temporally Contrastive Learning: An Alternative to the Laplacian in RL [140.12803111221206]
  In reinforcement learning, the graph Laplacian has proved to be a valuable tool in the task-agnostic setting. We propose an alternative method that is able to recover, in a non-uniform-prior setting, the expressiveness and the desired properties of the Laplacian representation. We find that our method succeeds as an alternative to the Laplacian in the non-uniform setting and scales to challenging continuous control environments.
  arXiv  Detail & Related papers  (2022-03-21T22:07:48Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching [13.100127636586317]
  We propose a method for learning the reward function (and the corresponding policy) to match the expert state density. We present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories.
  arXiv  Detail & Related papers  (2020-11-09T19:37:48Z)
• Regularized Inverse Reinforcement Learning [49.78352058771138]
  Inverse Reinforcement Learning (IRL) aims to facilitate a learner's ability to imitate expert behavior. Regularized IRL applies strongly convex regularizers to the learner's policy. We propose tractable solutions, and practical methods to obtain them, for regularized IRL.
  arXiv  Detail & Related papers  (2020-10-07T23:38:47Z)
• Continual Deep Learning by Functional Regularisation of Memorable Past [95.97578574330934]
  Continually learning new skills is important for intelligent systems, yet standard deep learning methods suffer from catastrophic forgetting of the past. We propose a new functional-regularisation approach that utilises a few memorable past examples crucial to avoid forgetting. Our method achieves state-of-the-art performance on standard benchmarks and opens a new direction for life-long learning where regularisation and memory-based methods are naturally combined.
  arXiv  Detail & Related papers  (2020-04-29T10:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.