Revisiting Glorot Initialization for Long-Range Linear Recurrences

• URL: http://arxiv.org/abs/2505.19827v1
• Date: Mon, 26 May 2025 11:04:59 GMT
• Title: Revisiting Glorot Initialization for Long-Range Linear Recurrences
• Abstract summary: Glorot is designed to ensure stable signal propagation.<n>Small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode.<n>We propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one.
• Score: 35.51491162256673
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation--but derived under the infinite-width, fixed-length regime--an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length $t = O(\sqrt{n})$, where $n$ is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.

Related papers

• Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity [23.519129043938758]
  Mirror Descent (MD) extends Gradient Descent (GD) beyond Euclidean geometry.<n>How sensitive are MD dynamics to their inputs?<n>For canonical KL-regularized MD on the simplex, we show that even linear objectives can amplify an initial $varepsilon$ perturbation exponentially fast.
  arXiv  Detail & Related papers  (2026-06-09T20:33:01Z)
• Learning Laplacian Eigenspace with Mass-Aware Neural Operators on Point Clouds [13.977358640919462]
  We introduce the Neuralspace Operator (NEO), a feed-forward framework to predict the spectrum directly from point clouds.<n>NEO circumvents the ill-posed nature of standard eigenvector regression, by learning the stable, invariant low-frequency subspace instead.<n>Our approach achieves near-linear runtime scaling and substantial wall-clock speedups over iterative solvers at comparable accuracy.
  arXiv  Detail & Related papers  (2026-05-23T04:21:20Z)
• Coverage Analysis of Rydberg Atom Quantum Receiver Arrays: A Stochastic Geometry Approach [55.47164192369501]
  Rydberg atomic quantum receivers (RAQRs) offer quantum-limited sensitivity and broadband tunability.<n>In dense deployments, aggregate interference can push atomic transducer out of its small-signal regime.<n>This paper addresses the question by embedding the RAQR front end into a geometry coverage analysis.
  arXiv  Detail & Related papers  (2026-05-22T04:05:13Z)
• A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks [0.0]
  In "reservoir computing", the weights of a readout are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics.<n>In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly layer model.<n>Here we derive a simple criterion to estimate the critical $g_c$ for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task.
  arXiv  Detail & Related papers  (2026-05-11T14:35:22Z)
• How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences [2.335152769484957]
  We study signal propagation in linear recurrent models at finite width.<n>We find that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones.
  arXiv  Detail & Related papers  (2026-05-06T16:44:44Z)
• The Procrustean Bed of Time Series: The Optimization Bias of Point-wise Loss [53.542743390809356]
  This paper aims to provide a first-principles analysis of the Expectation of Optimization Bias (EOB)<n>Our analysis reveals a fundamental paradigm paradox: the more deterministic and structured the time series, the more severe the bias by point-wise loss function.<n>We present a concrete solution that simultaneously achieves both principles via DFT or DWT.
  arXiv  Detail & Related papers  (2025-12-21T06:08:22Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization [58.851514333119255]
  This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in deep linear networks. We show that for all depth greater than one, with the standard Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters.
  arXiv  Detail & Related papers  (2023-06-22T23:14:57Z)
• Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations [17.51364577113718]
  We show three principles for architecture selection in finite width graph neural networks (GNNs) with ReLU activations. First, we theoretically derive what is essentially the unique generalization to ReLU GNNs of the well-known He-initialization. Second, we prove in finite width vanilla ReLU GNNs that oversmoothing is unavoidable at large depth when using fixed aggregation operator.
  arXiv  Detail & Related papers  (2023-06-20T16:40:41Z)
• Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK [86.45209429863858]
  We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime. We show that the neural networks possess a different limiting kernel which we call textitbias-generalized NTK We also study various properties of the neural networks with this new kernel.
  arXiv  Detail & Related papers  (2023-01-01T02:11:39Z)
• Rank-1 Matrix Completion with Gradient Descent and Small Random Initialization [12.783783498844022]
  The non completion formulation the truth matrix regular convergence problem has received significant attention in recent years.<n>In this paper, we prove and prove GD-1 converges to the ground trajectory small trajectory.<n>We provide an upper trajectory size that is sufficient to guarantee convergence.
  arXiv  Detail & Related papers  (2022-12-19T12:05:37Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Implicit Bias of Linear RNNs [27.41989861342218]
  Linear recurrent neural networks (RNNs) do not perform well on tasks requiring long-term memory. This paper provides a rigorous explanation of this property in the special case of linear RNNs. Using recently-developed kernel regime analysis, our main result shows that linear RNNs are functionally equivalent to a certain weighted 1D-convolutional network.
  arXiv  Detail & Related papers  (2021-01-19T19:39:28Z)
• Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization [79.42778415729475]
  We explore an alternative solution based on explicit memorization using linear autoencoders for sequences. We show how such pretraining can better support solving hard classification tasks with long sequences. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.
  arXiv  Detail & Related papers  (2020-11-05T14:57:16Z)
• Early stopping and polynomial smoothing in regression with reproducing kernels [2.0411082897313984]
  We study the problem of early stopping for iterative learning algorithms in a reproducing kernel Hilbert space (RKHS) We present a data-driven rule to perform early stopping without a validation set that is based on the so-called minimum discrepancy principle. The proposed rule is proved to be minimax-optimal over different types of kernel spaces.
  arXiv  Detail & Related papers  (2020-07-14T05:27:18Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)
• Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks [15.499554384036673]
  Batch normalization is an effective strategy to avoid rank collapse for both linear and ReLU networks. We derive a meaningful lower rank bound in deep linear networks. Empirically, we also demonstrate that this rank robustness generalizes to ReLU nets.
  arXiv  Detail & Related papers  (2020-03-03T17:21:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.