Hyperparameter Transfer in Graph Neural Networks

• URL: http://arxiv.org/abs/2607.05017v1
• Date: Mon, 06 Jul 2026 12:59:18 GMT
• Title: Hyperparameter Transfer in Graph Neural Networks
• Abstract summary: We develop and validate a transfer parameterization for graph neural networks (GNNs) trained with SGD, Adam, and AdamW.<n>We show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase.<n>These results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.
• Score: 8.584932159968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay. Hyperparameter transfer aims to make near-optimal hyperparameter settings consistent across model scale, so that large models can be optimized by proxy tuning their smaller, cheaper-to-optimize counterparts. While transfer principles are well-studied in the context of dense neural networks in language and vision tasks, they remain comparatively under-explored for graph neural networks (GNNs). We develop and validate a transfer parameterization for GNNs trained with SGD, Adam, and AdamW. Through theoretical scaling analyses and controlled experiments, we show that the proposed parameterization yields stable feature updates, learning rate transfer, and improved performance as width and depth increase. For SGD, we identify graph-dependent first-layer correction factors and show that their use can accelerate early training in graphs with sparse bag-of-words inputs. For Adam, we explore how different message passing normalizations affect early- and late-training transfer behavior, illustrating the importance of message passing normalization and advocating for an associated hyperparameter. For AdamW, we adapt a parameterization that allows for the joint transfer of weight decay and learning rate. Together, these results provide a practical recipe for scaling GNNs across a variety of learning tasks and training scenarios.

Related papers

• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• GQA-μP: The maximal parameterization update for grouped query attention [64.99972873396649]
  We show how to derivation P scalings for grouped-query attention (GQA)<n>We also demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter.
  arXiv  Detail & Related papers  (2026-05-14T18:03:16Z)
• Function-Space Learning Rates [23.09717258810923]
  We develop efficient methods to measure and set function-space learning rates in arbitrary neural networks.<n>We demonstrate two key applications: (1) analysing the dynamics of standard neural network optimisers in function space, rather than parameter space.
  arXiv  Detail & Related papers  (2025-02-24T18:31:58Z)
• UNEM: UNrolled Generalized EM for Transductive Few-Shot Learning [35.62208317531141]
  We advocate and introduce the unrolling paradigm, also referred to as "learning to optimize"<n>Our unrolling approach covers various statistical feature distributions and pre-training paradigms.<n>We report comprehensive experiments, which cover a breadth of fine-grained downstream image classification tasks.
  arXiv  Detail & Related papers  (2024-12-21T19:01:57Z)
• Efficient Hyperparameter Importance Assessment for CNNs [1.7778609937758323]
  This paper aims to quantify the importance weights of some hyperparameters in Convolutional Neural Networks (CNNs) with an algorithm called N-RReliefF. We conduct an extensive study by training over ten thousand CNN models across ten popular image classification datasets.
  arXiv  Detail & Related papers  (2024-10-11T15:47:46Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias [72.33336385797944]
  We propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias.<n>We show that LD significantly outperforms state-of-the-art methods on Open Graph Benchmark datasets.
  arXiv  Detail & Related papers  (2023-09-26T13:09:43Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• Transfer Learning with Convolutional Networks for Atmospheric Parameter Retrieval [14.131127382785973]
  The Infrared Atmospheric Sounding Interferometer (IASI) on board the MetOp satellite series provides important measurements for Numerical Weather Prediction (NWP) Retrieving accurate atmospheric parameters from the raw data provided by IASI is a large challenge, but necessary in order to use the data in NWP models. We show how features extracted from the IASI data by a CNN trained to predict a physical variable can be used as inputs to another statistical method designed to predict a different physical variable at low altitude.
  arXiv  Detail & Related papers  (2020-12-09T09:28:42Z)
• Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
  We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem) AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient. Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
  arXiv  Detail & Related papers  (2020-10-21T14:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.