On the Effective Horizon of Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2307.06541v2
• Date: Wed, 16 Oct 2024 16:59:58 GMT
• Title: On the Effective Horizon of Inverse Reinforcement Learning
• Authors: Yiqing Xu, Finale Doshi-Velez, David Hsu,
• Abstract summary: Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning over a given time horizon. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms.
• Score: 38.7571680927719
• License:
• Abstract: Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning over a given time horizon to compute an approximately optimal policy for a hypothesized reward function and then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an \emph{effective time horizon} shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis serves as a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results confirm the theoretical analysis.

Related papers

• Learning Uncertainty-Aware Temporally-Extended Actions [22.901453123868674]
  We propose a novel algorithm named Uncertainty-aware Temporal Extension (UTE) UTE employs ensemble methods to accurately measure uncertainty during action extension. We demonstrate the effectiveness of UTE through experiments in Gridworld and Atari 2600 environments.
  arXiv  Detail & Related papers  (2024-02-08T06:32:06Z)
• The Virtues of Pessimism in Inverse Reinforcement Learning [38.98656220917943]
  Inverse Reinforcement Learning is a powerful framework for learning complex behaviors from expert demonstrations. It is desirable to reduce the exploration burden by leveraging expert demonstrations in the inner-loop RL. We consider an alternative approach to speeding up the RL in IRL: emphpessimism, i.e., staying close to the expert's data distribution, instantiated via the use of offline RL algorithms.
  arXiv  Detail & Related papers  (2024-02-04T21:22:29Z)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective [55.36819597141271]
  Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an emphexpert policy -- plays a critical role in developing intelligent systems. This paper provides the first line of efficient IRL in vanilla offline and online settings using samples and runtime. As an application, we show that the learned rewards can emphtransfer to another target MDP with suitable guarantees.
  arXiv  Detail & Related papers  (2023-11-29T00:09:01Z)
• DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing [60.21269454707625]
  DreamSmooth learns to predict a temporally-smoothed reward, instead of the exact reward at the given timestep. We show that DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks.
  arXiv  Detail & Related papers  (2023-11-02T17:57:38Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• f-IRL: Inverse Reinforcement Learning via State Marginal Matching [13.100127636586317]
  We propose a method for learning the reward function (and the corresponding policy) to match the expert state density. We present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories.
  arXiv  Detail & Related papers  (2020-11-09T19:37:48Z)
• Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration [143.43658264904863]
  We show how iteration under a more standard notion of low inherent Bellman error, typically employed in least-square value-style algorithms, can provide strong PAC guarantees on learning a near optimal value function. We present a computationally tractable algorithm for the reward-free setting and show how it can be used to learn a near optimal policy for any (linear) reward function.
  arXiv  Detail & Related papers  (2020-08-18T04:34:21Z)
• Preference-based Reinforcement Learning with Finite-Time Guarantees [76.88632321436472]
  Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning to better elicit human opinion on the target objective. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. We present the first finite-time analysis for general PbRL problems.
  arXiv  Detail & Related papers  (2020-06-16T03:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.