Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors

• URL: http://arxiv.org/abs/2512.17180v2
• Date: Tue, 23 Dec 2025 06:26:45 GMT
• Title: Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors
• Abstract summary: This paper reveals an unexpected phenomenon in interactive reinforcement learning (IRL)<n>When given a choice between teachers with different reward structures, learning agents overwhelmingly prefer conservative, low-reward teachers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interactive reinforcement learning (IRL) has shown promise in enabling autonomous agents and robots to learn complex behaviours from human teachers, yet the dynamics of teacher selection remain poorly understood. This paper reveals an unexpected phenomenon in IRL: when given a choice between teachers with different reward structures, learning agents overwhelmingly prefer conservative, low-reward teachers (93.16% selection rate) over those offering 20x higher rewards. Through 1,250 experimental runs in navigation tasks with multiple expert teachers, we discovered: (1) Conservative bias dominates teacher selection: agents systematically choose the lowest-reward teacher, prioritising consistency over optimality; (2) Critical performance thresholds exist at teacher availability rho >= 0.6 and accuracy omega >= 0.6, below which the framework fails catastrophically; (3) The framework achieves 159% improvement over baseline Q-learning under concept drift. These findings challenge fundamental assumptions about optimal teaching in RL and suggest potential implications for human-robot collaboration, where human preferences for safety and consistency may align with the observed agent selection behaviour, potentially informing training paradigms for safety-critical robotic applications.

Related papers

• Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold [4.090731563364332]
  We argue that reinforcement learning policies should be viewed as local probes of a multi-basin reasoning solution manifold.<n>We propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation.
  arXiv  Detail & Related papers  (2026-07-30T07:05:37Z)
• Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams (Extended Version) [3.248039323542496]
  Preference-based reward learning casts the human teacher as a passive oracle answering learner-generated queries.<n>A teacher who knows the target can construct training examples more efficiently than any learner-driven acquisition strategy.<n>We recast preference learning as a human-autonomy team problem coupling two behavioral models.
  arXiv  Detail & Related papers  (2026-07-29T18:44:01Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents [50.78688247330029]
  On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students.<n>We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout.<n>Guided-OPD improves Score by 21.1% and Success Rate by 25.5% over vanilla OPD on average.
  arXiv  Detail & Related papers  (2026-06-14T16:41:45Z)
• When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning [45.79647925282674]
  On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher.<n>Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy.<n>We propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD.
  arXiv  Detail & Related papers  (2026-05-20T18:14:03Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment [82.00769536768509]
  Rank-Surprisal Ratio is a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory.<n>We demonstrate its practical utility in both trajectory selection and teacher selection.
  arXiv  Detail & Related papers  (2026-01-20T18:58:10Z)
• "The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework [16.96094045628127]
  Chain-of-Thought (CoT) reasoning empowers Large Language Models (LLMs) with remarkable capabilities but typically requires prohibitive parameter scales.<n>CoT distillation has emerged as a promising paradigm to transfer reasoning prowess into compact Student Models (SLMs)<n>We introduce COMPACT, a framework that adaptively fuses supervisions from different teachers by dynamically weighting teacher gradients.
  arXiv  Detail & Related papers  (2026-01-20T14:05:19Z)
• Interactive Distillation for Cooperative Multi-Agent Reinforcement Learning [0.49703640686206074]
  HINT (Hierarchical INteractive Teacher-based transfer) is a novel KD framework for MARL in a centralized training, decentralized execution setup.<n>By leveraging hierarchical RL, HINT provides a scalable, high-performing teacher.<n>We evaluate HINT on challenging cooperative domains (e.g., FireCommander for resource allocation, MARINE for tactical combat)
  arXiv  Detail & Related papers  (2026-01-08T22:16:43Z)
• Hierarchical Pedagogical Oversight: A Multi-Agent Adversarial Framework for Reliable AI Tutoring [0.0]
  We introduce Hierarchical Pedagogical Oversight (HPO), a framework that adapts structured adversarial synthesis to educational assessment.<n>Unlike cooperative multi-agent systems that often drift toward superficial consensus, HPO enforces a dialectical separation of concerns.<n>We evaluate this framework on the MRBench dataset of 1,214 middle-school mathematics dialogues.
  arXiv  Detail & Related papers  (2025-12-27T06:42:07Z)
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration [103.1589018460702]
  "guidance-on-demand" approach expands exploration while preserving the value of self-discovery.<n>Experiments show AMPO substantially outperforms a strong baseline.<n>Using four peer-sized teachers, our method achieves comparable results to approaches that leverage a single, more powerful teacher.
  arXiv  Detail & Related papers  (2025-10-02T17:14:00Z)
• Active teacher selection for reinforcement learning from human feedback [12.823935200868846]
  Reinforcement learning from human feedback (RLHF) enables machine learning systems to learn objectives from human feedback.<n>We propose the Hidden Utility Bandit framework to model differences in teacher rationality, expertise, and costliness.<n>We develop a variety of solution algorithms and apply them to two real-world domains: paper recommendation systems and COVID-19 vaccine testing.
  arXiv  Detail & Related papers  (2023-10-23T18:54:43Z)
• TGRL: An Algorithm for Teacher Guided Reinforcement Learning [45.38447023752256]
  It is common to train a policy to maximize a combination of reinforcement and teacher-student learning objectives. We present a $textitprincipled$ approach, along with an approximate implementation for $textitdynamically$ and $textitautomatically$ balancing when to follow the teacher and when to use rewards.
  arXiv  Detail & Related papers  (2023-07-06T17:58:40Z)
• Guarded Policy Optimization with Imperfect Online Demonstrations [32.22880650876471]
  Teacher-Student Framework is a reinforcement learning setting where a teacher agent guards the training of a student agent. It is expensive or even impossible to obtain a well-performing teacher policy. We develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance.
  arXiv  Detail & Related papers  (2023-03-03T06:24:04Z)
• Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation [70.92135839545314]
  We propose the dynamic prior knowledge (DPK), which integrates part of teacher's features as the prior knowledge before the feature distillation. Our DPK makes the performance of the student model positively correlated with that of the teacher model, which means that we can further boost the accuracy of students by applying larger teachers.
  arXiv  Detail & Related papers  (2022-06-13T11:52:13Z)
• Faculty Distillation with Optimal Transport [53.69235109551099]
  We propose to link teacher's task and student's task by optimal transport. Based on the semantic relationship between their label spaces, we can bridge the support gap between output distributions. Experiments under various settings demonstrate the succinctness and versatility of our method.
  arXiv  Detail & Related papers  (2022-04-25T09:34:37Z)
• Self-Training with Differentiable Teacher [80.62757989797095]
  Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. We propose ours, short for differentiable self-training, that treats teacher-student as a Stackelberg game.
  arXiv  Detail & Related papers  (2021-09-15T02:06:13Z)
• Dual Policy Distillation [58.43610940026261]
  Policy distillation, which transfers a teacher policy to a student policy, has achieved great success in challenging tasks of deep reinforcement learning. In this work, we introduce dual policy distillation(DPD), a student-student framework in which two learners operate on the same environment to explore different perspectives of the environment. The key challenge in developing this dual learning framework is to identify the beneficial knowledge from the peer learner for contemporary learning-based reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2020-06-07T06:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.