Conditioned Initialization for Attention

• URL: http://arxiv.org/abs/2609.07086v1
• Date: Mon, 07 Sep 2026 06:17:58 GMT
• Title: Conditioned Initialization for Attention
• Abstract summary: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond.<n>At the core of their success lies the attention layer, where the query, key, and value matrices determine how token dependencies are captured.<n>While considerable work has focused on scaling and optimizing Transformers, comparatively little attention has been paid to how the weights of the queries, keys and values are.
• Score: 32.435722001652174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their success lies the attention layer, where the query, key, and value matrices determine how token dependencies are captured. While considerable work has focused on scaling and optimizing Transformers, comparatively little attention has been paid to how the weights of the queries, keys and values are initialized. Common practice relies on random initialization or alternatives such as mimetic initialization, which imitates weight patterns from converged models, and weight selection, which transfers weights from a teacher model. In this paper, we argue that initialization can introduce an optimization bias that fundamentally shapes training dynamics. We propose conditioned initialization, a principled scheme that initializes attention weights to improve the spectral properties of the attention layer. Theoretically, we show that conditioned initialization can potentially reduce the condition number of the attention Jacobian, leading to more stable optimization. Empirically, it accelerates convergence and improves generalization across diverse applications, highlighting conditioning as a critical yet underexplored area for advancing Transformer performance. Importantly, conditioned initialization is simple to apply and integrates seamlessly into a wide range of Transformer architectures.

Related papers

• Small Initialization Matters for Large Language Models [28.726627416564842]
  We show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity.<n>Reducing the scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks.<n>We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling.
  arXiv  Detail & Related papers  (2026-06-16T13:53:48Z)
• A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks [0.0]
  In "reservoir computing", the weights of a readout are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics.<n>In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly layer model.<n>Here we derive a simple criterion to estimate the critical $g_c$ for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task.
  arXiv  Detail & Related papers  (2026-05-11T14:35:22Z)
• ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints [64.35580479051208]
  In previous works, low-rank adapters (LoRA) are randomly with a fixed rank across all attachment points.<n>In this paper, we improve convergence and final performance of LoRA fine-tuning using our proposed data-driven weight initialization method.
  arXiv  Detail & Related papers  (2025-07-09T23:52:31Z)
• The Primacy of Magnitude in Low-Rank Adaptation [15.583380841988868]
  Low-Rank Adaptation (LoRA) offers a parameter-efficient paradigm for tuning large models.<n>We propose LoRAM, a magnitude-driven "Basis & Basis" scheme that matches spectral methods without their inefficiencies.
  arXiv  Detail & Related papers  (2025-07-09T05:25:24Z)
• Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme [0.0]
  Emergence in machine learning refers to the spontaneous appearance of capabilities that arise from the scale and structure of training data.<n>We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence.<n>We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization.
  arXiv  Detail & Related papers  (2024-07-26T18:56:47Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• On the Convergence of Encoder-only Shallow Transformers [62.639819460956176]
  We build the global convergence theory of encoder-only shallow Transformers under a realistic setting. Our results can pave the way for a better understanding of modern Transformers, particularly on training dynamics.
  arXiv  Detail & Related papers  (2023-11-02T20:03:05Z)
• Geometry-aware training of factorized layers in tensor Tucker format [6.701651480567394]
  We introduce a novel approach to train the factors of a Tucker decomposition of the weight tensors. Our training proposal proves to be optimal in locally approximating the original unfactorized dynamics. We provide a theoretical analysis of the algorithm, showing convergence, approximation and local descent guarantees.
  arXiv  Detail & Related papers  (2023-05-30T14:20:51Z)
• Towards Theoretically Inspired Neural Initialization Optimization [66.04735385415427]
  We propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. We show that both the training and test performance of a network can be improved by maximizing GradCosine under norm constraint. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost.
  arXiv  Detail & Related papers  (2022-10-12T06:49:16Z)
• Data-driven Weight Initialization with Sylvester Solvers [72.11163104763071]
  We propose a data-driven scheme to initialize the parameters of a deep neural network. We show that our proposed method is especially effective in few-shot and fine-tuning settings.
  arXiv  Detail & Related papers  (2021-05-02T07:33:16Z)
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training [59.160154997555956]
  We present GradInit, an automated and architecture method for initializing neural networks. It is based on a simple agnostic; the variance of each network layer is adjusted so that a single step of SGD or Adam results in the smallest possible loss value. It also enables training the original Post-LN Transformer for machine translation without learning rate warmup.
  arXiv  Detail & Related papers  (2021-02-16T11:45:35Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.