Exponentially many initializations to avoid barren plateaus

• URL: http://arxiv.org/abs/2606.18515v1
• Date: Tue, 16 Jun 2026 22:07:17 GMT
• Title: Exponentially many initializations to avoid barren plateaus
• Abstract summary: Barren plateaus are stated as an average-case phenomenon: pick an ansatz, initialize it naively, and concentration follows.<n>We show that a potential cure for barren plateaus is simply to initialize the parameters more carefully.
• Score: 0.6282171844772423
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Barren plateaus are stated as an average-case phenomenon: pick an ansatz, initialize it naively, and concentration follows. This has led to the common view that a potential cure for barren plateaus is simply to initialize the parameters more carefully. Here we show that the situation is subtler. We introduce a first-moment framework that gives a simple operator-level diagnostic for when an initialization may escape the fully concentrated barren-plateau fixed point, and for comparing the biases induced by different initialization strategies. Our framework recovers several known initialization schemes such as identity and Gaussian initialization, but also shows that barren-plateau avoidance is highly non-unique. Indeed, many shifted, biased, and non-symmetric parameter distributions can avoid concentration, and these choices need not be equivalent. In fact, our results show that one can generate exponentially many families of inequivalent initialization strategies. Then, our numerics indicate that different first-moment-distinct initializations can lead to different attained minima, suggesting that avoiding barren plateaus via smart initializations can trade the exponential concentration problem for the challenge of selecting the right trainable pocket amongst many options.

Related papers

• Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry [52.75288960194919]
  Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models.<n>We introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit.
  arXiv  Detail & Related papers  (2026-09-09T22:35:31Z)
• Probably Approximately Precision and Recall Learning [60.00180898830079]
  A key challenge in machine learning is the prevalence of one-sided feedback.<n>We introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels.<n>We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Semi-Supervised Unconstrained Head Pose Estimation in the Wild [57.11798881492183]
  We propose the first semi-supervised unconstrained head pose estimation method SemiUHPE.<n>Our method is based on the observation that the aspect-ratio invariant cropping of wild heads is superior to previous landmark-based affine alignment.<n>Our proposed method is also beneficial for solving other closely related problems, including generic object rotation regression and 3D head reconstruction.
  arXiv  Detail & Related papers  (2024-04-03T08:01:00Z)
• Semi-Supervised Laplace Learning on Stiefel Manifolds [48.3427853588646]
  We develop the framework Sequential Subspace for graph-based, supervised samples at low-label rates. We achieves that our methods at extremely low rates, and high label rates.
  arXiv  Detail & Related papers  (2023-07-31T20:19:36Z)
• Restarts subject to approximate sharpness: A parameter-free and optimal scheme for first-order methods [0.6554326244334866]
  Sharpness is an assumption in continuous optimization that bounds the distance from minima by objective function suboptimality. Sharpness involves problem-specific constants that are typically unknown, and restart schemes typically reduce convergence rates. We consider the assumption of approximate sharpness, a generalization of sharpness that incorporates an unknown constant perturbation to the objective function error.
  arXiv  Detail & Related papers  (2023-01-05T19:01:41Z)
• BEINIT: Avoiding Barren Plateaus in Variational Quantum Algorithms [0.7462336024223667]
  Barren plateaus are a notorious problem in the optimization of variational quantum algorithms. We propose an alternative strategy which initializes the parameters of a unitary gate by drawing from a beta distribution. We empirically show that our proposed framework significantly reduces the possibility of a complex quantum neural network getting stuck in a barren plateau.
  arXiv  Detail & Related papers  (2022-04-28T19:46:10Z)
• E-detectors: a nonparametric framework for sequential change detection [86.15115654324488]
  We develop a fundamentally new and general framework for sequential change detection. Our procedures come with clean, nonasymptotic bounds on the average run length. We show how to design their mixtures in order to achieve both statistical and computational efficiency.
  arXiv  Detail & Related papers  (2022-03-07T17:25:02Z)
• Towards strong pruning for lottery tickets with non-zero biases [6.85316573653194]
  Lottery ticket hypothesis holds promise that pruning randomly deep neural networks could offer efficient alternative to deep learning. Common parameter schemes and existence proofs are focused on networks with zero gradient biases. We extend these schemes and existence proofs to non-zero biases, including explicit 'looks-linear' approaches for ReLU activation functions.
  arXiv  Detail & Related papers  (2021-10-21T13:56:04Z)
• On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent [55.96478231566129]
  We show that relative scales play an important role in determining the learned model. We develop a technique for deriving the inductive bias of gradient-flow.
  arXiv  Detail & Related papers  (2021-02-19T07:10:48Z)
• Renormalization for Initialization of Rolling Shutter Visual-Inertial Odometry [5.33024001730262]
  Initialization is a prerequisite for using inertial signals and fusing them with visual data. We propose a novel statistical solution to the problem on visual and inertial data simultaneously, by casting it into the renormalization scheme of Kanatani. Extensive evaluations on ground truth exhibit superior performance and a gain in accuracy of up to $20%$ over the originally proposed Least Squares solution.
  arXiv  Detail & Related papers  (2020-08-14T14:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.