Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)

• URL: http://arxiv.org/abs/2605.05742v1
• Date: Thu, 07 May 2026 06:33:25 GMT
• Title: Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)
• Abstract summary: We show that weak-to-strong generalization occurs in standard linear logistic regression.<n>We show that this happens for most student-teacher pairs, suggesting that weak-to-strong generalization is in fact emphalmost inevitable, even in this basic setting.
• Score: 7.661703938597629
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Weak-to-strong generalization is a phenomenon in post-training whereby a strong student model, when finetuned solely with feedback from a weaker teacher, can not only surpass the teacher, but can improve upon its own capabilities. Recent work of Burns et al. (2023) demonstrated that this can occur in the setting of frontier language models, and subsequently there has been a flurry of both empirical work trying to exploit this phenomenon, as well as theoretical work attempting to understand it. In this work, we demonstrate that weak-to-strong generalization occurs in standard linear logistic regression, under mild distributional assumptions on the data. In fact, we show that this happens for most student-teacher pairs, suggesting that weak-to-strong generalization is in fact \emph{almost inevitable}, even in this basic setting. Notably, our setting does not require the student to be more expressive or have more model capacity in any way compared to the teacher, which runs contrary to the prevailing theoretical belief that a mismatch in model capacity is a central mechanism to weak-to-strong generalization.

Related papers

• On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective [28.005935031887038]
  Weak-to-strong generalization, where a student model trained on imperfect labels surpasses that teacher, has been widely observed.<n>In this paper, through a theoretical analysis of simple models, we uncover three core mechanisms that can drive this phenomenon.
  arXiv  Detail & Related papers  (2025-05-23T20:09:09Z)
• How to Mitigate Overfitting in Weak-to-strong Generalization? [50.37526669608372]
  Weak-to-strong generalization aims to elicit the capabilities of strong models through weak supervisors.<n>Strong models exhibit significant overfitting in weak-to-strong generalization.<n>We propose a two-stage framework that simultaneously improves the quality of supervision signals and the quality of input questions.
  arXiv  Detail & Related papers  (2025-03-06T09:32:39Z)
• Weak-to-Strong Generalization Even in Random Feature Networks, Provably [52.6431973649838]
  We show that weak-to-strong generalization does not require a strong learner like GPT-4.<n>We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the weak teacher.
  arXiv  Detail & Related papers  (2025-03-04T18:58:00Z)
• Deep Learning is Not So Mysterious or Different [54.5330466151362]
  We argue that anomalous generalization behaviour is not distinct to neural networks.<n>We present soft inductive biases as a key unifying principle in explaining these phenomena.<n>We also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning.
  arXiv  Detail & Related papers  (2025-03-03T22:56:04Z)
• The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration [38.49808553313377]
  We provide theoretical insights into weak-to-strong generalization.<n>We show that the weak model should demonstrate strong generalization performance and maintain well-calibrated predictions.<n>We extend the work of Charikar et al. (2024) to a loss function based on Kullback-Leibler divergence.
  arXiv  Detail & Related papers  (2025-02-03T15:48:28Z)
• Provable Weak-to-Strong Generalization via Benign Overfitting [3.4652800888823294]
  We consider the inverted situation, where a weak teacher supervises a strong student with imperfect pseudolabels.<n>We theoretically investigate weak-to-strong generalization for binary and multilabel classification.<n>Our techniques should eventually extend to weak-to-strong multiclass classification.
  arXiv  Detail & Related papers  (2024-10-06T22:10:50Z)
• Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization [68.62228569439478]
  We investigate whether there exists an issue of weak-to-strong deception.<n>We find that the deception intensifies as the capability gap between weak and strong models increases.<n>Our work highlights the urgent need to pay more attention to the true reliability of superalignment.
  arXiv  Detail & Related papers  (2024-06-17T11:36:39Z)
• Theoretical Analysis of Weak-to-Strong Generalization [23.235671743867492]
  We show that existing weak supervision theory fails to account for pseudolabel correction and coverage expansion. Our bounds capture the intuition that weak-to-strong generalization occurs when the strong model is unable to fit the mistakes of the weak teacher without incurring additional error.
  arXiv  Detail & Related papers  (2024-05-25T03:48:12Z)
• Quantifying the Gain in Weak-to-Strong Generalization [14.453654853392619]
  We show that the improvement in performance achieved by strong models over their weaker counterparts is quantified by the misfit error incurred by the strong model on labels generated by the weaker model. For instance, we can predict the amount by which the strong model will improve over the weak model, and also choose among different weak models to train the strong model, based on its misfit error.
  arXiv  Detail & Related papers  (2024-05-24T00:14:16Z)
• On the Generalization Ability of Unsupervised Pretraining [53.06175754026037]
  Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. This paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase. Our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.
  arXiv  Detail & Related papers  (2024-03-11T16:23:42Z)
• Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Models [55.919653720979824]
  This paper focuses on the concept of weak-to-strong generalization, which involves using a weaker model to supervise a stronger one. We introduce a novel and adaptively adjustable loss function for weak-to-strong supervision. Our approach not only exceeds the performance benchmarks set by strong-to-strong generalization but also surpasses the outcomes of fine-tuning strong models with whole datasets.
  arXiv  Detail & Related papers  (2024-02-06T06:30:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.