On the Effective Horizon of Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2307.06541v3
• Date: Thu, 20 Feb 2025 05:56:10 GMT
• Title: On the Effective Horizon of Inverse Reinforcement Learning
• Authors: Yiqing Xu, Finale Doshi-Velez, David Hsu,
• Abstract summary: Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning, over a given time horizon. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. This work formally analyzes this phenomenon and provides an explanation.
• Score: 38.7571680927719
• License:
• Abstract: Inverse reinforcement learning (IRL) algorithms often rely on (forward) reinforcement learning or planning, over a given time horizon, to compute an approximately optimal policy for a hypothesized reward function; they then match this policy with expert demonstrations. The time horizon plays a critical role in determining both the accuracy of reward estimates and the computational efficiency of IRL algorithms. Interestingly, an *effective time horizon* shorter than the ground-truth value often produces better results faster. This work formally analyzes this phenomenon and provides an explanation: the time horizon controls the complexity of an induced policy class and mitigates overfitting with limited data. This analysis provides a guide for the principled choice of the effective horizon for IRL. It also prompts us to re-examine the classic IRL formulation: it is more natural to learn jointly the reward and the effective horizon rather than the reward alone with a given horizon. To validate our findings, we implement a cross-validation extension and the experimental results support the theoretical analysis. The project page and code are publicly available.

Related papers

• In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates [10.438810967483438]
  Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy. Current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. This paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory.
  arXiv  Detail & Related papers  (2024-10-21T03:16:32Z)
• The Virtues of Pessimism in Inverse Reinforcement Learning [38.98656220917943]
  Inverse Reinforcement Learning is a powerful framework for learning complex behaviors from expert demonstrations. It is desirable to reduce the exploration burden by leveraging expert demonstrations in the inner-loop RL. We consider an alternative approach to speeding up the RL in IRL: emphpessimism, i.e., staying close to the expert's data distribution, instantiated via the use of offline RL algorithms.
  arXiv  Detail & Related papers  (2024-02-04T21:22:29Z)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective [55.36819597141271]
  Inverse Reinforcement Learning (IRL) -- the problem of learning reward functions from demonstrations of an emphexpert policy -- plays a critical role in developing intelligent systems. This paper provides the first line of efficient IRL in vanilla offline and online settings using samples and runtime. As an application, we show that the learned rewards can emphtransfer to another target MDP with suitable guarantees.
  arXiv  Detail & Related papers  (2023-11-29T00:09:01Z)
• DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing [60.21269454707625]
  DreamSmooth learns to predict a temporally-smoothed reward, instead of the exact reward at the given timestep. We show that DreamSmooth achieves state-of-the-art performance on long-horizon sparse-reward tasks.
  arXiv  Detail & Related papers  (2023-11-02T17:57:38Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration [143.43658264904863]
  We show how iteration under a more standard notion of low inherent Bellman error, typically employed in least-square value-style algorithms, can provide strong PAC guarantees on learning a near optimal value function. We present a computationally tractable algorithm for the reward-free setting and show how it can be used to learn a near optimal policy for any (linear) reward function.
  arXiv  Detail & Related papers  (2020-08-18T04:34:21Z)
• Preference-based Reinforcement Learning with Finite-Time Guarantees [76.88632321436472]
  Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning to better elicit human opinion on the target objective. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. We present the first finite-time analysis for general PbRL problems.
  arXiv  Detail & Related papers  (2020-06-16T03:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.