Across the Loss Landscape with Progressive Growth

• URL: http://arxiv.org/abs/2608.24568v1
• Date: Tue, 25 Aug 2026 13:50:48 GMT
• Title: Across the Loss Landscape with Progressive Growth
• Abstract summary: We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
• Score: 51.366966420881646
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

Related papers

• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks [22.81761236732655]
  We study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.<n>Our analysis tracks the trajectory through three phases: an alignment phase in which hidden weights progressively align with the planted direction while the output weights maintain the correct sign pattern.<n>We establish novel uniform concentration results that hold along the entire trajectory, and are essential for obtaining order-wise optimal sample complexity.
  arXiv  Detail & Related papers  (2026-06-03T05:44:30Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations [53.348574336527854]
  We investigate the neural network map as a function on the space of weights and biases.<n>We prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures.
  arXiv  Detail & Related papers  (2025-10-28T14:34:33Z)
• Description of the Training Process of Neural Networks via Ergodic Theorem : Ghost nodes [3.637162892228131]
  We present a unified framework for understanding and accelerating deep neural networks via training gradient descent (SGD)<n>We introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which distinguishes genuine convergence toward stablers.<n>We propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions.
  arXiv  Detail & Related papers  (2025-07-01T17:54:35Z)
• Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Bounds [49.1574468325115]
  First-order adaptive optimization methods like Adam are the default choices for training modern deep neural networks.<n>We develop a multi-layer refinement framework that progressively tightens bounds on region crossings.<n>We prove that the number of region crossings collapses from exponential to near-linear in the effective dimension.
  arXiv  Detail & Related papers  (2025-05-21T01:34:16Z)
• Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape [41.94295877935867]
  We present a new conceptual framework to understand how curvature of expected changes in loss emerges in architectures with many rectified linear units. Our derivations show how these discontinuities combine to form a glass-like structure, similar to amorphous solids that contain microscopic domains of strong, but random, atomic alignment. We derive the optimal modification to quasi-Newton steps that incorporate both glass and Hessian terms, as well as certain exactness properties that are possible with Nesterov-accelerated gradient updates.
  arXiv  Detail & Related papers  (2024-11-25T20:32:57Z)
• Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction [49.66486092259376]
  The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift. Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures. We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
  arXiv  Detail & Related papers  (2023-06-12T16:28:11Z)
• Implicit Sparse Regularization: The Impact of Depth and Early Stopping [35.4113861165802]
  We show that early stopping is crucial for gradient descent to converge to a sparse model. We characterize the impact of depth and early stopping and show that for a general depth parameter N, gradient descent with early stopping achieves minimax optimal sparse recovery.
  arXiv  Detail & Related papers  (2021-08-12T07:43:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.