On the Asymptotics of Self-Supervised Pre-training: Two-Stage M-Estimation and Representation Symmetry

• URL: http://arxiv.org/abs/2603.27631v1
• Date: Sun, 29 Mar 2026 11:04:03 GMT
• Title: On the Asymptotics of Self-Supervised Pre-training: Two-Stage M-Estimation and Representation Symmetry
• Abstract summary: Self-supervised pre-training has become a cornerstone of modern machine learning.<n>Existing bounds leave open the question of how sharp the current rates are.<n>We develop an intrinsic theory of pre-training via two-stage M-estimation.
• Score: 8.92071749364712
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised pre-training, where large corpora of unlabeled data are used to learn representations for downstream fine-tuning, has become a cornerstone of modern machine learning. While a growing body of theoretical work has begun to analyze this paradigm, existing bounds leave open the question of how sharp the current rates are, and whether they accurately capture the complex interaction between pre-training and fine-tuning. In this paper, we address this gap by developing an asymptotic theory of pre-training via two-stage M-estimation. A key challenge is that the pre-training estimator is often identifiable only up to a group symmetry, a feature common in representation learning that requires careful treatment. We address this issue using tools from Riemannian geometry to study the intrinsic parameters of the pre-training representation, which we link with the downstream predictor through a notion of orbit-invariance, precisely characterizing the limiting distribution of the downstream test risk. We apply our main result to several case studies, including spectral pre-training, factor models, and Gaussian mixture models, and obtain substantial improvements in problem-specific factors over prior art when applicable.

Related papers

• Unified Inference Framework for Single and Multi-Player Performative Prediction: Method and Asymptotic Optimality [15.289993502701305]
  This paper introduces a unified statistical inference framework that bridges single-agent and multi-agent performativity.<n>It provides a principled toolkit for reliable estimation and decision-making in dynamic, performative environments.
  arXiv  Detail & Related papers  (2026-02-03T03:17:54Z)
• Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity [49.809923981964715]
  Contaminated mixture of experts (MoE) is motivated by transfer learning methods where a pre-trained model, acting as a frozen expert, is integrated with an adapter model, functioning as a trainable expert, in order to learn a new task.<n>In this work, we characterize uniform convergence rates for estimating parameters under challenging settings where ground-truth parameters vary with the sample size.<n>We also establish corresponding minimax lower bounds to ensure that these rates are minimax optimal.
  arXiv  Detail & Related papers  (2026-01-31T23:45:50Z)
• Asymptotic Theory of Iterated Empirical Risk Minimization, with Applications to Active Learning [15.858234832499585]
  We study a class of iterated empirical risk (ERM) procedures in which two successive ERMs are performed on the same dataset.<n>For linear models trained with a broad class of convex losses on Gaussian mixture data, we derive a sharp characterization of the test error.<n>We uncover a fundamental tradeoff in how the labeling budget should be allocated across stages, and demonstrate a double-descent behavior of the test error driven purely by data selection.
  arXiv  Detail & Related papers  (2026-01-30T14:39:51Z)
• The Coverage Principle: How Pre-Training Enables Post-Training [70.25788947586297]
  We study how pre-training shapes the success of the final model.<n>We uncover a mechanism that explains the power of coverage in predicting downstream performance.
  arXiv  Detail & Related papers  (2025-10-16T17:53:50Z)
• Incorporating priors in learning: a random matrix study under a teacher-student framework [6.744353807473373]
  Regularized linear regression is central to machine learning, yet its high-dimensional behavior with informative priors remains poorly understood.<n>We provide the first exact characterization of training and test risks for maximum a posteriori (MAP) regression.<n>Our framework unifies ridge regression, least squares, and prior-informed estimators, and -- using random matrix theory -- yields closed-form risk formulas.
  arXiv  Detail & Related papers  (2025-09-26T09:47:15Z)
• Analyzing Generalization in Pre-Trained Symbolic Regression [17.789199791229624]
  Symbolic regression algorithms search a space of mathematical expressions for formulas that explain given data.<n> Transformer-based models have emerged as a promising, promising approach shifting the expensive search to a large-scale pre-training phase.
  arXiv  Detail & Related papers  (2025-09-24T07:47:02Z)
• Multiply Robust Conformal Risk Control with Coarsened Data [0.0]
  Conformal Prediction (CP) has recently received a tremendous amount of interest.<n>In this paper, we consider the general problem of obtaining distribution-free valid prediction regions for an outcome given coarsened data.<n>Our principled use of semiparametric theory has the key advantage of facilitating flexible machine learning methods.
  arXiv  Detail & Related papers  (2025-08-21T12:14:44Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.