Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression

• URL: http://arxiv.org/abs/2603.05691v1
• Date: Thu, 05 Mar 2026 21:32:59 GMT
• Title: Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression
• Abstract summary: We show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error.<n>Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher.
• Score: 27.0471650610245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained from a weak teacher, and yet the strong student outperforms the weak teacher. In this paper, we show that the potential improvement is substantial, in the sense that it affects the scaling law followed by the test error. Specifically, we consider students and teachers trained via random feature ridge regression (RFRR). Our main technical contribution is to derive a deterministic equivalent for the excess test error of the student trained on labels obtained via the teacher. Via this deterministic equivalent, we then identify regimes in which the scaling law of the student improves upon that of the teacher, unveiling that the improvement can be achieved both in bias-dominated and variance-dominated settings. Strikingly, the student may attain the minimax optimal rate regardless of the scaling law of the teacher -- in fact, when the test error of the teacher does not even decay with the sample size.

Related papers

• Coupled Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback [24.756084698191568]
  Large language model (LLM) distillation aims to transfer the capabilities of a powerful teacher to a smaller student.<n>Direct imitation, however, can also transfer the teacher's systematic bias and errors.<n>We propose Coupled and Learning (CCL), an algorithm that couples teacher calibration with student updates.
  arXiv  Detail & Related papers  (2026-09-15T17:15:40Z)
• A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation [59.43605629311855]
  We analyze the gradient of the per-token K2 estimator of reverse KL with respect to the student logits.<n>Our primary contribution is therefore a gradient-level characterization of reverse-KL OPD trained with the K2 estimator, with SuRe as one empirical instantiation.
  arXiv  Detail & Related papers  (2026-08-26T11:14:42Z)
• Prediction-Only Distillation in Linear and Logistic Regression [61.07540493350384]
  Self-distillation is typically studied when the student is retrained on the teacher's original training inputs.<n>We study SD in this prediction-only regime through a fresh-X prediction-mixed scheme.<n>For binary logistic regression, we show that prediction mixing can outperform both the teacher and the pure-distilled.
  arXiv  Detail & Related papers  (2026-07-16T20:45:56Z)
• Toward Understanding Adversarial Distillation: Why Robust Teachers Fail [22.989324947501018]
  Adrial Distillation aims to enhance student robustness by guiding the student with a robust teacher's soft labels.<n>We identify a key mechanism of this teacher dependency: the misalignment between the teacher's supervisory confidence and the student's representational limitations.
  arXiv  Detail & Related papers  (2026-05-21T04:57:15Z)
• Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation [14.526961596109361]
  We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model.<n>A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM)<n>We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions.
  arXiv  Detail & Related papers  (2026-03-26T14:07:47Z)
• Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment [82.00769536768509]
  Rank-Surprisal Ratio is a simple metric that captures both alignment and informativeness to assess the suitability of a reasoning trajectory.<n>We demonstrate its practical utility in both trajectory selection and teacher selection.
  arXiv  Detail & Related papers  (2026-01-20T18:58:10Z)
• Provable Weak-to-Strong Generalization via Benign Overfitting [3.4652800888823294]
  We consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels.<n>We theoretically investigate weak-to-strong generalization for binary and multilabel classification.<n>Our techniques should eventually extend to weak-to-strong multiclass classification.
  arXiv  Detail & Related papers  (2024-10-06T22:10:50Z)
• Theoretical Analysis of Weak-to-Strong Generalization [23.235671743867492]
  We show that existing weak supervision theory fails to account for pseudolabel correction and coverage expansion. Our bounds capture the intuition that weak-to-strong generalization occurs when the strong model is unable to fit the mistakes of the weak teacher without incurring additional error.
  arXiv  Detail & Related papers  (2024-05-25T03:48:12Z)
• On student-teacher deviations in distillation: does it pay to disobey? [54.908344098305804]
  Knowledge distillation has been widely used to improve the test accuracy of a "student" network. Despite being trained to fit the teacher's probabilities, the student may not only significantly deviate from the teacher probabilities, but may also outdo the teacher in performance.
  arXiv  Detail & Related papers  (2023-01-30T14:25:02Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Self-Training with Differentiable Teacher [80.62757989797095]
  Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. We propose ours, short for differentiable self-training, that treats teacher-student as a Stackelberg game.
  arXiv  Detail & Related papers  (2021-09-15T02:06:13Z)
• Teacher's pet: understanding and mitigating biases in distillation [61.44867470297283]
  Several works have shown that distillation significantly boosts the student's overall performance. However, are these gains uniform across all data subgroups? We show that distillation can harm performance on certain subgroups. We present techniques which soften the teacher influence for subgroups where it is less reliable.
  arXiv  Detail & Related papers  (2021-06-19T13:06:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.