Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension

• URL: http://arxiv.org/abs/2607.24887v1
• Date: Mon, 27 Jul 2026 12:46:19 GMT
• Title: Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension
• Abstract summary: We introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients.<n>By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability.<n> Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment.
• Score: 13.776610961556202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal-noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train-test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Finite-Sample Metric Non-Collapse for Geometrically Supervised Latent World Models in Control [0.0]
  We establish a finite-sample learning-to-control theory for geometrically supervised latent models of nonlinear deterministic systems.<n>We introduce an encoder-only local--global metric hinge that enforces directional resolution and separated-state discrimination.<n>A modular deterministic transfers the learned certificates to trajectory, finite-horizon cost, learned-cost-head, and corollary guarantees, while a validated finite-net result enables sharper model-specific certification.
  arXiv  Detail & Related papers  (2026-08-07T14:25:33Z)
• Quantifying Uncertainty In Wide Two-Layer Neural Networks: On The Law Of The Limiting Fluctuation Process [3.0171175820634826]
  Uncertainty quantification in neural networks prediction is a main issue for usual applications.<n>Our approach seeks at reducing costs by directly evaluating uncertainty using PDE's information on the variance.<n>We thus study the law of the limiting process describing the random fluctuations around the mean-field limit of wide two-layer neural networks trained by gradient descent.
  arXiv  Detail & Related papers  (2026-06-04T10:25:23Z)
• Trainability Beyond Linearity in Variational Quantum Objectives [0.0]
  We show that the objective itself admits a fixed-observable representation if and only if loss is affine.<n>We show that the amplification-capable objective produces resolved several orders of magnitude larger than affine.
  arXiv  Detail & Related papers  (2026-04-20T21:12:10Z)
• LACE: Loss-Adaptive Capacity Expansion for Continual Learning [0.0]
  LACE (Loss-Adaptive Capacity Expansion) is an online mechanism that expands a model's representational capacity during training.<n>LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.
  arXiv  Detail & Related papers  (2026-03-30T15:58:33Z)
• Leverage-Weighted Conformal Prediction [1.3537117504260623]
  Split conformal prediction provides distribution-free prediction intervals with finite-sample marginal coverage.<n>We propose Leverage-Weighted Conformal Prediction, which weights nonconformity scores by a function of the statistical leverage.
  arXiv  Detail & Related papers  (2026-02-13T07:49:39Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Neural Optimal Transport Meets Multivariate Conformal Prediction [58.43397908730771]
  We propose a framework for conditional vectorile regression (CVQR)<n>CVQR combines neural optimal transport with quantized optimization, and apply it to predictions.
  arXiv  Detail & Related papers  (2025-09-29T19:50:19Z)
• Modes of Sequence Models and Learning Coefficients [0.6906005491572401]
  We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks.<n>We show theoretically that Local Learning Coefficient estimates are insensitive to modes below a data-dependent threshold.<n>This insight clarifies why reliable LLC estimates can be obtained even when a network parameter is not a strict minimiser of the population loss.
  arXiv  Detail & Related papers  (2025-04-25T03:38:10Z)
• A New Formulation of Lipschitz Constrained With Functional Gradient Learning for GANs [52.55025869932486]
  This paper introduces a promising alternative method for training Generative Adversarial Networks (GANs) on large-scale datasets with clear theoretical guarantees.<n>We propose a novel Lipschitz-constrained Functional Gradient GANs learning (Li-CFG) method to stabilize the training of GAN.<n>We demonstrate that the neighborhood size of the latent vector can be reduced by increasing the norm of the discriminator gradient.
  arXiv  Detail & Related papers  (2025-01-20T02:48:07Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Near-optimal inference in adaptive linear regression [60.08422051718195]
  Even simple methods like least squares can exhibit non-normal behavior when data is collected in an adaptive manner. We propose a family of online debiasing estimators to correct these distributional anomalies in at least squares estimation. We demonstrate the usefulness of our theory via applications to multi-armed bandit, autoregressive time series estimation, and active learning with exploration.
  arXiv  Detail & Related papers  (2021-07-05T21:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.