Teacher-Student Representational Alignment for Reinforcement Learning-Driven Imitation Learning

• URL: http://arxiv.org/abs/2605.28372v1
• Date: Wed, 27 May 2026 12:12:54 GMT
• Title: Teacher-Student Representational Alignment for Reinforcement Learning-Driven Imitation Learning
• Abstract summary: Imitation learning from a state-based reinforcement learning (RL) policy is a common approach to overcome the curse of dimensionality in complex and high-dimensional observation spaces prevalent in robotics.<n>This paper addresses the irreducible imitation gap that emerges when teacher and student are learned in isolation.<n>We propose a novel algorithm which learns a shared embedding space that hides agent-specific observations and thus trains imitable teacher policies by construction.
• Score: 11.538345159297839
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation learning (IL) from a state-based reinforcement learning (RL) policy is a common approach to overcome the curse of dimensionality in complex and high-dimensional observation spaces prevalent in robotics. This paper addresses the irreducible imitation gap that emerges when teacher and student are learned in isolation, and the teacher policy has the liberty to rely on privileged state information that the student cannot infer from its observations. Instead of improving poor student performance with RL finetuning after IL, which often requires a whole new training setup, we propose a novel algorithm which learns a shared embedding space that hides agent-specific observations and thus trains imitable teacher policies by construction. We train the shared embedding space with self-supervised contrastive learning in parallel to the teacher policy and prevent it from extracting private information by limiting its gradients from updating the encoder networks. We perform evaluations on several example domains and compare to state-of-the-art baselines showing that our algorithm enables higher student performance with substantially reduced imitation gap.

Related papers

• dOPSD: On-Policy Self-Distillation for Diffusion Language Models [52.60302464420127]
  Diffusion large language models generate text by iteratively denoising a masked sequence.<n>Supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards.<n>On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher.
  arXiv  Detail & Related papers  (2026-07-05T17:47:29Z)
• Teacher-Guided Policy Optimization for LLM Distillation [90.49982387646861]
  Teacher-Guided Policy Optimization (TGPO) is an on-policy algorithm that incorporates dense directional guidance by leveraging teacher predictions conditioned on the student's rollout.<n>Experiments on complex reasoning benchmarks demonstrate that TGPO significantly outperforms standard baselines and is robust to different teachers.
  arXiv  Detail & Related papers  (2026-05-13T09:20:03Z)
• Decentralized Learning Strategies for Estimation Error Minimization with Graph Neural Networks [86.99017195607077]
  We address real-time sampling and estimation of autoregressive Markovian sources in wireless networks.<n>We propose a graphical reinforcement learning framework for policy optimization.<n>Theoretically, our proposed policies are transferable, allowing a policy trained on one graph to be effectively applied to structurally similar graphs.
  arXiv  Detail & Related papers  (2026-01-19T02:18:45Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotion [3.1194372040101928]
  Quadrupedal locomotion via Reinforcement Learning (RL) is commonly addressed using the teacher-student paradigm.<n>We propose Teacher-Aligned Representations via Contrastive Learning (TAR), a framework that leverages privileged information with self-supervised contrastive learning.<n>Results showed accelerated training by 2x compared to state-of-the-art baselines to achieve peak performance.
  arXiv  Detail & Related papers  (2025-03-26T12:49:26Z)
• Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning [26.53136644321385]
  Imitation learning (IL) aims to mimic the behavior of an expert in a sequential decision making task by learning from demonstrations.<n>We show that it is possible to achieve horizon-independent sample complexity in offline IL whenever the range of the cumulative payoffs is controlled.<n>Specializing in deterministic, stationary policies, we show that the gap between offline and online IL is smaller than previously thought.
  arXiv  Detail & Related papers  (2024-07-20T23:31:56Z)
• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Dealing with the Unknown: Pessimistic Offline Reinforcement Learning [25.30634466168587]
  We propose a Pessimistic Offline Reinforcement Learning (PessORL) algorithm to actively lead the agent back to the area where it is familiar. We focus on problems caused by out-of-distribution (OOD) states, and deliberately penalize high values at states that are absent in the training dataset.
  arXiv  Detail & Related papers  (2021-11-09T22:38:58Z)
• Privacy-Preserving Teacher-Student Deep Reinforcement Learning [23.934121758649052]
  We develop a private mechanism that protects the privacy of the teacher's training dataset. We empirically show that the algorithm improves the student's learning upon convergence rate and utility.
  arXiv  Detail & Related papers  (2021-02-18T20:15:09Z)
• Bridging the Imitation Gap by Adaptive Insubordination [88.35564081175642]
  We show that when the teaching agent makes decisions with access to privileged information, this information is marginalized during imitation learning. We propose 'Adaptive Insubordination' (ADVISOR) to address this gap. ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration.
  arXiv  Detail & Related papers  (2020-07-23T17:59:57Z)
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch [14.334987432342707]
  We propose a new framework for transfer learning where the teacher and the student can have arbitrarily different state- and action-spaces. To handle this mismatch, we produce embeddings which can systematically extract knowledge from the teacher policy and value networks. We demonstrate successful transfer learning in situations when the teacher and student have different state- and action-spaces.
  arXiv  Detail & Related papers  (2020-06-12T09:51:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.