GQA-μP: The maximal parameterization update for grouped query attention

• URL: http://arxiv.org/abs/2605.15290v1
• Date: Thu, 14 May 2026 18:03:16 GMT
• Title: GQA-μP: The maximal parameterization update for grouped query attention
• Abstract summary: We show how to derivation P scalings for grouped-query attention (GQA)<n>We also demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter.
• Score: 64.99972873396649
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization (μP) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of Yang et al. (2023a), we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μP scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay.

Related papers

• Hyperparameter Transfer in Graph Neural Networks [8.584932159968]
  We develop and validate a transfer parameterization for graph neural networks (GNNs) trained with SGD, Adam, and AdamW.<n>We show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase.<n>These results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.
  arXiv  Detail & Related papers  (2026-07-06T12:59:18Z)
• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate [10.599439539657787]
  We show that Maximal Update ($$P) offers high-quality learning rate transfer relative to standard parameterization.<n>We find that the overwhelming benefit of $$P relative to standard parameterization arises simply from maximizing the learning rate of the embedding layer.
  arXiv  Detail & Related papers  (2026-05-20T17:59:40Z)
• Learning parameter curves in feedback-based quantum optimization algorithms [0.0]
  We train a teacher-student model to map a MaxCut problem instance to an associated FQA parameter curve.<n> Numerical experiments show that this model can accurately predict FQA parameter curves across a range of problem sizes.<n>These results suggest that machine learning can offer a practical path to reducing sampling costs and resource overheads in quantum algorithms.
  arXiv  Detail & Related papers  (2026-01-13T00:03:44Z)
• High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>We present textbfSMoA, a high-rank textbfStructured textbfMOdulation textbfAdapter that uses fewer trainable parameters while maintaining a higher rank.
  arXiv  Detail & Related papers  (2026-01-12T13:06:17Z)
• Quantum-Informed Machine Learning for Predicting Spatiotemporal Chaos [1.5216516276847551]
  We introduce a quantum-informed machine learning (QIML) framework for the long-term dynamical behavior of high-dimensional chaotic systems.<n>We evaluate QIML on three representative systems: the Kuramoto-Sivashinsky equation, the two-dimensional Kolmogorov flow, and a cross-section of fully developed three-dimensional turbulent channel flow.
  arXiv  Detail & Related papers  (2025-07-26T08:36:16Z)
• RSQ: Learning from Important Tokens Leads to Better Quantized LLMs [65.5558181902098]
  Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining.<n>We propose RSQ (Rotate, Scale, then Quantize), which applies rotations to the model to mitigate outliers.<n>We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and three model families.
  arXiv  Detail & Related papers  (2025-03-03T18:46:33Z)
• NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search [7.971065005161565]
  quantization is a technique to convert floating point representations to low bit-width fixed point representations. We show how to learn new quantized weights over the entire quantized space. We show the ability of the method to achieve state-of-the-art compression rates in both, data-free and data-driven configurations.
  arXiv  Detail & Related papers  (2023-08-10T14:19:58Z)
• Weight Re-Mapping for Variational Quantum Algorithms [54.854986762287126]
  We introduce the concept of weight re-mapping for variational quantum circuits (VQCs) We employ seven distinct weight re-mapping functions to assess their impact on eight classification datasets. Our results indicate that weight re-mapping can enhance the convergence speed of the VQC.
  arXiv  Detail & Related papers  (2023-06-09T09:42:21Z)
• Pre-training Tensor-Train Networks Facilitates Machine Learning with Variational Quantum Circuits [70.97518416003358]
  Variational quantum circuits (VQCs) hold promise for quantum machine learning on noisy intermediate-scale quantum (NISQ) devices. While tensor-train networks (TTNs) can enhance VQC representation and generalization, the resulting hybrid model, TTN-VQC, faces optimization challenges due to the Polyak-Lojasiewicz (PL) condition. To mitigate this challenge, we introduce Pre+TTN-VQC, a pre-trained TTN model combined with a VQC.
  arXiv  Detail & Related papers  (2023-05-18T03:08:18Z)
• Improving Convergence for Quantum Variational Classifiers using Weight Re-Mapping [60.086820254217336]
  In recent years, quantum machine learning has seen a substantial increase in the use of variational quantum circuits (VQCs) We introduce weight re-mapping for VQCs, to unambiguously map the weights to an interval of length $2pi$. We demonstrate that weight re-mapping increased test accuracy for the Wine dataset by $10%$ over using unmodified weights.
  arXiv  Detail & Related papers  (2022-12-22T13:23:19Z)
• Flatter, faster: scaling momentum for optimal speedup of SGD [0.0]
  We study training dynamics arising from interplay between gradient descent (SGD) and label noise and momentum in the training of neural networks. We find that scaling the momentum hyper parameter $1-NISTbeta$ with the learning rate to the power of $2/3$ maximally accelerates training, without sacrificing generalization.
  arXiv  Detail & Related papers  (2022-10-28T20:41:48Z)
• NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer [45.47667026025716]
  We propose a novel, robust and accelerated iteration that relies on two key elements. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively. We show that NAG-arity is competitive with state-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models.
  arXiv  Detail & Related papers  (2022-09-29T16:54:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.