A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks

• URL: http://arxiv.org/abs/2605.10650v1
• Date: Mon, 11 May 2026 14:35:22 GMT
• Title: A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks
• Abstract summary: In "reservoir computing", the weights of a readout are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics.<n>In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly layer model.<n>Here we derive a simple criterion to estimate the critical $g_c$ for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Proper weight initialization prior to training has historically been one of the key factors that helped kick off the deep learning revolution. Initialization is even more crucial in "reservoir computing", where the weights of a readout layer are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics. In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly initialized model. The phase transition is controlled by the weight variance $g^2$ and separates an ordered phase from a chaotic one where information progressively degrades. Here we derive a simple criterion to estimate the critical $g_c$ for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task. Finally, we argue that our criterion can serve as a design principle for future initialization schemes.

Related papers

• Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry [52.75288960194919]
  Recent work has identified incremental learning in shallow networks trained on single-index and multi-index models.<n>We introduce a symmetry-based finite-width approximation via symmetrized networks, rather than comparing directly with an infinite-width limit.
  arXiv  Detail & Related papers  (2026-09-09T22:35:31Z)
• Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias [0.5405981353784005]
  We show that low-learning-rate SGD can interpolate while still remembering its initialization.<n>We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve memory, whereas finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization.
  arXiv  Detail & Related papers  (2026-05-27T22:30:39Z)
• A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning [51.505728136705564]
  We develop an analytical theory of the pretraining-fine-tuning pipeline in diagonal linear networks.<n>We find that different initialization choices place the network into four distinct fine-tuning regimes.<n>A smaller scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization.
  arXiv  Detail & Related papers  (2026-02-23T17:19:33Z)
• ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints [64.35580479051208]
  In previous works, low-rank adapters (LoRA) are randomly with a fixed rank across all attachment points.<n>In this paper, we improve convergence and final performance of LoRA fine-tuning using our proposed data-driven weight initialization method.
  arXiv  Detail & Related papers  (2025-07-09T23:52:31Z)
• Sinusoidal Initialization, Time for a New Start [0.5242869847419834]
  Initialization plays a critical role in Deep Neural Network training, directly influencing convergence, stability, and generalization.<n>We introduce a novel deterministic method that employs sinusoidal functions to construct structured weight matrices to improve the spread and balance of weights throughout the network.<n>Our experiments show an increase of 4.9% in final validation accuracy and 20.9% in convergence speed.
  arXiv  Detail & Related papers  (2025-05-19T09:45:18Z)
• Initialization Matters for Adversarial Transfer Learning [61.89451332757625]
  We discover the necessity of an adversarially robust pretrained model. We propose Robust Linear Initialization (RoLI) for adversarial finetuning, which initializes the linear head with the weights obtained by adversarial linear probing. Across five different image classification datasets, we demonstrate the effectiveness of RoLI and achieve new state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-10T00:51:05Z)
• Scaling and Resizing Symmetry in Feedforward Networks [0.0]
  We show that the scaling property exhibited by physical systems at criticality, is also present in untrained feedforward networks with random weights at the critical line. We suggest an additional data-resizing symmetry, which is directly inherited from the scaling symmetry at criticality.
  arXiv  Detail & Related papers  (2023-06-26T18:55:54Z)
• On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent [55.96478231566129]
  We show that relative scales play an important role in determining the learned model. We develop a technique for deriving the inductive bias of gradient-flow.
  arXiv  Detail & Related papers  (2021-02-19T07:10:48Z)
• Structured Weight Priors for Convolutional Neural Networks [74.1348917619643]
  This paper explores the benefits of adding structure to weight priors. It first considers first-layer filters of a convolutional NN, designing a prior based on random Gabor filters. Empirical results suggest that these structured weight priors lead to more meaningful functional priors for image data.
  arXiv  Detail & Related papers  (2020-07-12T13:05:51Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.