Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization

• URL: http://arxiv.org/abs/2605.08131v1
• Date: Fri, 01 May 2026 05:01:10 GMT
• Title: Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization
• Abstract summary: Inverse reinforcement learning (IRL) learns a reward function and a corresponding policy that best fit the demonstration data of an expert.<n>In the current IRL setting, the learner is isolated from the expert and can only passively observe the expert demonstrations.<n>This paper studies interactive IRL (IIRL) where a learner aims to learn the reward function of an expert and a policy to interact with the expert.
• Score: 35.515246456075054
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverse reinforcement learning (IRL) learns a reward function and a corresponding policy that best fit the demonstration data of an expert. However, in the current IRL setting, the learner is isolated from the expert and can only passively observe the expert demonstrations. This limits the applicability of IRL to interactive settings, where the learner actively interacts with the expert and needs to infer the expert's reward function from the interactions. To bridge the gap, this paper studies interactive IRL (IIRL) where a learner aims to learn the reward function of an expert and a policy to interact with the expert during its interactions with the expert. We formulate IIRL as a stochastic bi-level optimization problem where the lower level learns a reward function to explain the behaviors of the expert, and the upper level learns a policy to interact with the expert. We develop a double-loop algorithm, Bi-level Interactive Scenarios Inverse Reinforcement Learning (BISIRL), which solves the lower-level problem in the inner loop and the upper-level problem in the outer loop. We formally guarantee that BISIRL converges and validate our algorithm through extensive experiments.

Related papers

• When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning [70.72643569128316]
  Imitation learning (IL) training an agent to replicate expert behavior from demonstrations.<n>We introduce OVI, an interactive on-policy IL algorithm that is statistically efficient whenever the learner can represent the expert's value function.<n>OVI outperforms offline policy-based (BC), interactive policy-based (DAgger), and offline value-based IL methods, with the largest gains when the learner network is substantially less expressive than the expert's.
  arXiv  Detail & Related papers  (2026-07-31T16:52:47Z)
• Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching [23.600285251963395]
  In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment.<n>Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimize the reward through repeated RL procedures.<n>We propose a novel approach to IRL by direct policy optimization, exploiting a linear factorization of the return as the inner product of successor features and a reward vector.
  arXiv  Detail & Related papers  (2024-11-11T14:05:50Z)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
  Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
  arXiv  Detail & Related papers  (2024-03-11T17:49:18Z)
• Inverse Reinforcement Learning with Sub-optimal Experts [56.553106680769474]
  We study the theoretical properties of the class of reward functions that are compatible with a given set of experts. Our results show that the presence of multiple sub-optimal experts can significantly shrink the set of compatible rewards. We analyze a uniform sampling algorithm that results in being minimax optimal whenever the sub-optimal experts' performance level is sufficiently close to the one of the optimal agent.
  arXiv  Detail & Related papers  (2024-01-08T12:39:25Z)
• Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms [22.703438243976876]
  We study interactive imitation learning, where a learner interactively queries a demonstrating expert for action annotations. We propose a new oracle-efficient algorithm MFTPL-P with provable finite-sample guarantees.
  arXiv  Detail & Related papers  (2023-12-28T07:05:30Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning [26.05184273238923]
  This work aims to tackle a major challenge in offline Inverse Reinforcement Learning (IRL) We devise a principled algorithm (namely CLARE) that solves offline IRL efficiently via integrating "conservatism" into a learned reward function. Our theoretical analysis provides an upper bound on the return gap between the learned policy and the expert policy.
  arXiv  Detail & Related papers  (2023-02-09T17:16:29Z)
• Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games [30.720112378448285]
  We formulate inverse reinforcement learning as an expert-learner interaction. The optimal performance intent of an expert or target agent is unknown to a learner agent. We develop an off-policy IRL algorithm that does not require knowledge of the expert and learner agent dynamics.
  arXiv  Detail & Related papers  (2023-01-05T10:35:08Z)
• Learning from Guided Play: Improving Exploration for Adversarial Imitation Learning with Simple Auxiliary Tasks [8.320969283401233]
  We show that the standard, naive approach to exploration can manifest as a suboptimal local maximum. We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of multiple exploratory, auxiliary tasks.
  arXiv  Detail & Related papers  (2022-12-30T20:38:54Z)
• Active Exploration for Inverse Reinforcement Learning [58.295273181096036]
  We propose a novel IRL algorithm: Active exploration for Inverse Reinforcement Learning (AceIRL) AceIRL actively explores an unknown environment and expert policy to quickly learn the expert's reward function and identify a good policy. We empirically evaluate AceIRL in simulations and find that it significantly outperforms more naive exploration strategies.
  arXiv  Detail & Related papers  (2022-07-18T14:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.