Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation

• URL: http://arxiv.org/abs/2603.25466v1
• Date: Thu, 26 Mar 2026 14:07:47 GMT
• Title: Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation
• Abstract summary: We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model.<n>A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM)<n>We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions.
• Score: 14.526961596109361
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM). This approach directly propagates any systematic bias or mis-specification present in the teacher, thereby degrading the student's predictions. We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions. Our analysis shows how the student can thereby emulate a proximal gradient scheme for solving an oracle optimization problem, and this provably reduces the effect of teacher bias. For general student--teacher pairs, we establish non-asymptotic excess risk bounds for any RaT fixed point, along with convergence guarantees for the student-teacher iterative scheme. For kernel-based student--teacher pairs, we prove a sharp separation: the RaT method achieves the minimax-optimal rate, while the SM method incurs constant prediction error for any sample size. Experiments on both synthetic data and ImageNette classification under covariate shift corroborate our theoretical findings.

Related papers

• Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback [24.756084698191568]
  Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student.<n>Direct imitation, however, can also transfer the teacher's systematic bias and errors.<n>We propose Coupled and Learning (CCL), an algorithm that couples teacher calibration with student updates.
  arXiv  Detail & Related papers  (2026-09-15T17:15:40Z)
• Prediction-Only Distillation in Linear and Logistic Regression [61.07540493350384]
  Self-distillation is typically studied when the student is retrained on the teacher's original training inputs.<n>We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme.<n>For binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled.
  arXiv  Detail & Related papers  (2026-07-16T20:45:56Z)
• On-Policy Distillation with Best-of-N Teacher Rollout Selection [54.91780727674628]
  We propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation.<n>BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the teacher trajectory.<n>BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets.
  arXiv  Detail & Related papers  (2026-05-10T19:49:00Z)
• Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression [27.0471650610245]
  We show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error.<n>Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher.
  arXiv  Detail & Related papers  (2026-03-05T21:32:59Z)
• Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment [82.00769536768509]
  Rank-Surprisal Ratio is a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory.<n>We demonstrate its practical utility in both trajectory selection and teacher selection.
  arXiv  Detail & Related papers  (2026-01-20T18:58:10Z)
• Long-Chain Reasoning Distillation via Adaptive Prefix Alignment [57.130176131042965]
  We propose a framework that exploits teacher CoTs for distillation through adaptive prefix alignment.<n>P-ALIGN adaptively truncates teacher-generated reasoning trajectories by determining whether the remaining suffix is concise.<n>Experiments on multiple mathematical reasoning benchmarks demonstrate that P-ALIGN outperforms all baselines by over 3%.
  arXiv  Detail & Related papers  (2026-01-15T04:40:45Z)
• On the Emergence of Weak-to-Strong Generalization: A Bias-Variance Perspective [25.571896278624024]
  Weak-to-strong generalization (W2SG) refers to the phenomenon where a strong student model, trained on a dataset labeled by a weak teacher, outperforms the teacher on the target task.<n>Recent studies attribute this performance gain to the prediction misfit between the student and teacher models.<n>We show that W2SG is more likely to emerge when the student model approximates its posterior mean teacher, rather than mimicking an individual teacher.
  arXiv  Detail & Related papers  (2025-05-30T07:52:43Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Toward Student-Oriented Teacher Network Training For Knowledge Distillation [40.55715466657349]
  We propose a teacher training method SoTeacher which incorporates Lipschitz regularization and consistency regularization into ERM. Experiments on benchmark datasets using various knowledge distillation algorithms and teacher-student pairs confirm that SoTeacher can improve student accuracy consistently.
  arXiv  Detail & Related papers  (2022-06-14T07:51:25Z)
• Graph Consistency based Mean-Teaching for Unsupervised Domain Adaptive Person Re-Identification [54.58165777717885]
  This paper proposes a Graph Consistency based Mean-Teaching (GCMT) method with constructing the Graph Consistency Constraint (GCC) between teacher and student networks. Experiments on three datasets, i.e., Market-1501, DukeMTMCreID, and MSMT17, show that proposed GCMT outperforms state-of-the-art methods by clear margin.
  arXiv  Detail & Related papers  (2021-05-11T04:09:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.