Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

• URL: http://arxiv.org/abs/2604.00258v1
• Date: Tue, 31 Mar 2026 21:34:18 GMT
• Title: Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards
• Abstract summary: We argue that imperfect student demonstrations are not noise to be discarded, but structured signals provided their relative quality is ranked.<n>We introduce HALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which ranks them within a hierarchical learning framework.<n>Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations.
• Score: 8.858171965804912
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert demonstrations under a fixed reward. Real-world student interactions, however, are often inherently imperfect and evolving: students explore, make errors, revise strategies, and refine their goals as understanding develops. In this work, we argue that imperfect student demonstrations are not noise to be discarded, but structured signals-provided their relative quality is ranked. We introduce HALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which not only leverages sub-optimal student demonstrations, but ranks them within a hierarchical learning framework. HALIDE models student behavior at multiple levels of abstraction, enabling inference of higher-level intent and strategy from suboptimal actions while explicitly capturing the temporal evolution of student reward functions. By integrating demonstration quality into hierarchical reward inference,HALIDE distinguishes transient errors from suboptimal strategies and meaningful progress toward higher-level learning goals. Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations.

Related papers

• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Imitation Learning via Focused Satisficing [6.745370992941109]
  Imitation learning assumes that demonstrations are close to optimal according to some fixed, but unknown, cost function.<n>We show experimentally that this focuses the policy to imitate the highest quality (portions of) demonstrations better than existing imitation learning methods.
  arXiv  Detail & Related papers  (2025-05-20T18:36:52Z)
• Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning [12.4468604987226]
  This paper introduces Learning to Discern (L2D), an offline imitation learning framework for learning from demonstrations with diverse quality and style. We show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy performance across a range of tasks in both simulations and on a physical robot.
  arXiv  Detail & Related papers  (2023-10-22T06:08:55Z)
• Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations [60.241144377865716]
  We consider the imitation of sub-optimal demonstrations, with both a small clean demonstration set and a large noisy set. We propose method by evaluating and imitating at the sub-demonstration level, encoding action primitives of varying quality into different skills.
  arXiv  Detail & Related papers  (2023-06-13T17:24:37Z)
• Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning [48.595574101874575]
  In the real world, expert demonstrations are more likely to be imperfect. A positive-unlabeled adversarial imitation learning algorithm is developed. Agent policy will be optimized to cheat the discriminator and produce trajectories similar to those optimal expert demonstrations.
  arXiv  Detail & Related papers  (2023-02-13T11:26:44Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Imitation Learning by Estimating Expertise of Demonstrators [92.20185160311036]
  We show that unsupervised learning over demonstrator expertise can lead to a consistent boost in the performance of imitation learning algorithms. We develop and optimize a joint model over a learned policy and expertise levels of the demonstrators. We illustrate our findings on real-robotic continuous control tasks from Robomimic and discrete environments such as MiniGrid and chess.
  arXiv  Detail & Related papers  (2022-02-02T21:23:19Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• HILONet: Hierarchical Imitation Learning from Non-Aligned Observations [8.258872189267045]
  It is challenging learning from demonstrated observation-only trajectories in a non-time-aligned environment. We propose a new imitation learning approach called Hierarchical Learning from Observation(HILONet), which adopts a hierarchical structure to choose feasible sub-goals.
  arXiv  Detail & Related papers  (2020-11-05T05:48:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.