On the Blessing of Pre-training in Weak-to-Strong Generalization

• URL: http://arxiv.org/abs/2605.05710v1
• Date: Thu, 07 May 2026 05:55:10 GMT
• Title: On the Blessing of Pre-training in Weak-to-Strong Generalization
• Abstract summary: We identify pre-training as the prerequisite for the emergence of Weak-to-Strong Generalization (W2SG)<n>We demonstrate that W2SG is not an innate capability but emerges via a phase transition tightly coupled with pre-training.
• Score: 38.765691061220686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paradigm of Weak-to-Strong Generalization (W2SG) suggests that a pre-trained strong model can surpass its weak supervisor, yet the decisive role of pre-training remains theoretically and empirically under-explored. In this work, we identify pre-training as the essential prerequisite for the emergence of W2SG. Theoretically, we formalize the W2SG problem within a high-dimensional single-index model framework using spiked Gaussian data, modeling pre-training as a spectral initialization step. Building upon prior impossibility results regarding the failure of learning under random initialization, we prove that W2SG is achievable when pre-training provides a geometric warm start that places the model within an "effective region" characterized by a perturbed strong-convexity geometry. Within this region, we derive a rigorous generalization bound that naturally captures the optimization dynamics: an initial performance improvement followed by a saturation bottleneck dictated by the weak supervisor's bias. Empirically, we first validate all our assumptions and theoretical insights through controlled synthetic simulations. Finally, through a massive-scale evaluation of hundreds of intermediate pre-training checkpoints from large language models, we demonstrate that W2SG is not an innate capability but emerges via a phase transition tightly coupled with the progression of pre-training.

Related papers

• Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition [45.191945465591374]
  Task-specific lightweight models for surgical phase recognition excel at capturing temporal dynamics but generalize poorly under domain shift.<n>We propose textbfLarge-textbfSmall textbfTemporal adaptation (textbfLaST), a novel large-small collaborative framework that enables zero-shot adaptation to unseen clinical domains.
  arXiv  Detail & Related papers  (2026-08-24T07:06:52Z)
• The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge [51.56484100374058]
  Weak-to-strong (W2S) generalization is proposed as an approach to aligning AI systems.<n>We study W2S in the setting of reward-model learning with two-layer neural networks.<n>We prove that the strong model efficiently learns task $$, eliciting its pre-trained knowledge while retaining general capabilities.
  arXiv  Detail & Related papers  (2026-05-13T02:35:58Z)
• S-GRPO: Unified Post-Training for Large Vision-Language Models [11.901218794806796]
  We propose Supervised Group Relative Policy Optimization (S-GRPO) for adapting Large Vision-Language Models (LVLMs)<n>S-GRPO integrates the guidance of imitation learning into the multi-trajectory exploration of preference optimization.<n>It gracefully bridges the gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)
  arXiv  Detail & Related papers  (2026-04-17T08:39:07Z)
• Out-of-Support Generalisation via Weight Space Sequence Modelling [1.6921396880325779]
  We reformulate the OoS generalisation problem as a sequence modelling task in the weight space.<n>Our framework yields plausible, interpretable, and uncertainty-aware predictions without necessitating explicit inductive biases.<n>These results hold significant implications for the wider adoption of artificial intelligence in safety-critical applications.
  arXiv  Detail & Related papers  (2026-02-14T01:51:54Z)
• Enhancing Generalization of Depth Estimation Foundation Model via Weakly-Supervised Adaptation with Regularization [21.788680301776207]
  We propose WeSTAR, a parameter-efficient framework that performs Weakly supervised Self-Training Adaptation with Regularization.<n>We first adopt a dense self-training objective as the primary source of structural self-supervision.<n>To further improve robustness, we introduce semantically-aware hierarchical normalization.
  arXiv  Detail & Related papers  (2025-11-18T08:16:16Z)
• Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models [26.393123295267642]
  Weak-to-Strong generalization (W2SG) is a new trend to elicit the full capabilities of a strong model with supervision from a weak model.<n>We fine-tune a strong model with trajectories of intermediate actions generated by a weak model.<n>Our empirical evaluations demonstrate substantial improvements in reasoning and decision-making capabilities across diverse task domains.
  arXiv  Detail & Related papers  (2025-07-25T00:17:09Z)
• On the Generalization Ability of Unsupervised Pretraining [53.06175754026037]
  Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. This paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase. Our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.
  arXiv  Detail & Related papers  (2024-03-11T16:23:42Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Stochastic Training is Not Necessary for Generalization [57.04880404584737]
  It is widely believed that the implicit regularization of gradient descent (SGD) is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve strong performance on CIFAR-10 that is on-par with SGD.
  arXiv  Detail & Related papers  (2021-09-29T00:50:00Z)
• On the Generalization of Stochastic Gradient Descent with Momentum [58.900860437254885]
  We first show that there exists a convex loss function for which algorithmic stability fails to establish generalization guarantees. For smooth Lipschitz loss functions, we analyze a modified momentum-based update rule, and show that it admits an upper-bound on the generalization error. For the special case of strongly convex loss functions, we find a range of momentum such that multiple epochs of standard SGDM, as a special form of SGDEM, also generalizes.
  arXiv  Detail & Related papers  (2021-02-26T18:58:29Z)
• On the Generalization of Stochastic Gradient Descent with Momentum [84.54924994010703]
  momentum-based accelerated variants of gradient descent (SGD) are widely used when training machine learning models. We first show that there exists a convex loss function for which the stability gap for multiple epochs of SGD with standard heavy-ball momentum (SGDM) becomes unbounded. For smooth Lipschitz loss functions, we analyze a modified momentum-based update rule, i.e., SGD with early momentum (SGDEM) under a broad range of step-sizes.
  arXiv  Detail & Related papers  (2018-09-12T17:02:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.