On the robustness of noisy solutions in non-convex neural networks

• URL: http://arxiv.org/abs/2607.27000v1
• Date: Wed, 29 Jul 2026 14:57:26 GMT
• Title: On the robustness of noisy solutions in non-convex neural networks
• Abstract summary: We show that frozen one-step replica-symmetry-breaking solution densities survive at any finite temperature.<n>We show that dense, algorithmically accessible regions of finite-energy persist beyond $_rm OGP$, up to a threshold $_rm OGP()$ that grows with the allowed error.<n>In the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization.
• Score: 42.548424354574585
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimization in non-convex neural network models is strongly influenced by the geometry of the solution space: sparse, isolated, point-like clusters are typically algorithmically inaccessible, whereas wide and flat regions can be found efficiently despite being relatively rare. At zero temperature this picture has been formalized in binary perceptrons through the overlap gap property (OGP), which limits algorithmic access to configurations with zero training error above a critical constraint density $α_{\rm OGP}$. Here we extend this description to finite temperature, where a positive training error is allowed and statistically penalized. We first show that the frozen one-step replica-symmetry-breaking solution, dominating the zero temperature equilibrium measure, survives at any finite temperature. We furthermore derive a general criterion, based on the smoothness of the single-pattern Gibbs weight near the decision boundary, that determines when a finite-temperature relaxation of the loss removes freezing. We then extend the OGP construction to finite temperature and show that dense, algorithmically accessible regions of finite-energy configurations persist beyond $α_{\rm OGP}$, up to a threshold $α_{\rm OGP}(ε)$ that grows with the allowed training error $ε$. Finally, in the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization. Using a finite energy message-passing algorithm, we demonstrate numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs [48.096969031315744]
  Previous work has established that for the RBF kernel and a certain range of bandwidths $r$, the prior degenerates in the limit.<n>We prove that for $r$ below the threshold $r_c(d)$ the prior converges to a limit distribution $_barZ$.
  arXiv  Detail & Related papers  (2026-06-06T15:12:43Z)
• A rigorous quasipolynomial-time classical algorithm for SYK thermal expectations [51.660331450043806]
  Estimating local observables in Gibbs states is a central problem in quantum simulation.<n>We give a proof of a quasipolynomial-time classical algorithm that estimates SYK local thermal expectations at sufficiently high constant temperature.<n>Our result introduces a new Wick-pair expansion that we expect to be broadly useful for quantum many-body systems.
  arXiv  Detail & Related papers  (2026-04-22T21:14:04Z)
• Finite-temperature quantum Krylov method from real-time overlaps [0.0]
  We introduce a real-time overlap sequence $g_n=langle |e-inH|rangle$, which enables thermodynamic quantities to be obtained over a broad temperature range.<n>For the one-dimensional spin-$frac12$ Heisenberg model, we obtain accurate specific heat, magnetic susceptibility, and entropy in the noiseless case.<n>Our results establish real-time-overlap-based finite-temperature evaluation as a promising framework for finite-temperature computation on near-future quantum hardware.
  arXiv  Detail & Related papers  (2026-04-12T09:16:46Z)
• Low-temperature Gibbs states with tensor networks [0.0]
  We introduce a tensor network method for approximating thermal equilibrium states of quantum many-body systems at low temperatures.<n>Motivated by properties of the ground state for conformal field theories, our ansatz is especially well suited near criticality.<n>We show how the ansatz reproduces the finite temperature scaling of entanglement in a conformal field theory.
  arXiv  Detail & Related papers  (2025-01-14T18:29:20Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Confinement in 1+1D $\mathbb{Z}_2$ Lattice Gauge Theories at Finite Temperature [0.0]
  We study confinement in a simple one-dimensional $mathbbZ$ lattice gauge theory at finite temperature and filling. Our results shed new light on confinement at finite temperature from an experimentally relevant standpoint.
  arXiv  Detail & Related papers  (2023-08-16T18:00:01Z)
• Efficient Simulation of Quantum Many-body Thermodynamics by Tailoring Zero-temperature Tensor Network [2.13230439190003]
  We propose to access the finite-temperature properties from the tensor network (TN) representing the zero-temperature partition function. The proposed idea can be extended to higher-dimensional systems of bosons and fermions.
  arXiv  Detail & Related papers  (2022-02-01T06:46:52Z)
• Erasure tolerant quantum memory and the quantum null energy condition in holographic systems [0.41998444721319217]
  Investigating principles for storage of quantum information at finite temperature with minimal need for active error correction is an active area of research. We study an explicit encoding of a logical qubit into two similar chirally propagating excitations of finite von-Neumann entropy on a finite temperature background. We show that the quantum null energy condition gives analytic results for the minimal finite temperature needed for the deletion.
  arXiv  Detail & Related papers  (2022-01-31T19:00:04Z)
• Uhlmann Fidelity and Fidelity Susceptibility for Integrable Spin Chains at Finite Temperature: Exact Results [68.8204255655161]
  We show that the proper inclusion of the odd parity subspace leads to the enhancement of maximal fidelity susceptibility in the intermediate range of temperatures. The correct low-temperature behavior is captured by an approximation involving the two lowest many-body energy eigenstates.
  arXiv  Detail & Related papers  (2021-05-11T14:08:02Z)
• Free Energy Wells and Overlap Gap Property in Sparse PCA [81.64027805404483]
  We study a variant of the sparse PCA (principal component analysis) problem in the "hard" regime. We show bounds on the depth of free energy wells for various Gibbs measures naturally associated to the problem. We prove that the Overlap Gap Property (OGP) holds in a significant part of the hard regime.
  arXiv  Detail & Related papers  (2020-06-18T17:18:02Z)
• Convex Geometry and Duality of Over-parameterized Neural Networks [70.15611146583068]
  We develop a convex analytic approach to analyze finite width two-layer ReLU networks. We show that an optimal solution to the regularized training problem can be characterized as extreme points of a convex set. In higher dimensions, we show that the training problem can be cast as a finite dimensional convex problem with infinitely many constraints.
  arXiv  Detail & Related papers  (2020-02-25T23:05:33Z)
• Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamics [7.1822457112352955]
  We consider the problem of function approximation by two-layer neural nets with random weights that are "nearly Gaussian" We show that the problem can be phrased as global minimization of a free energy functional on the space of paths over probability measures on the weights.
  arXiv  Detail & Related papers  (2020-02-05T20:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.