A Survey of Weight Space Learning: Understanding, Representation, and Generation

• URL: http://arxiv.org/abs/2603.10090v1
• Date: Tue, 10 Mar 2026 15:32:23 GMT
• Title: A Survey of Weight Space Learning: Understanding, Representation, and Generation
• Abstract summary: Weight Space Learning treats neural weights as a meaningful domain for analysis and modeling.<n>We categorize existing methods into three core dimensions: Weight Space Understanding, Weight Space Representation, and Weight Space Generation.<n>We show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction.
• Score: 87.2501236345188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

Related papers

• Position: Weight Space Should Be a First-Class Generative AI Modality [67.10702320971893]
  This paper argues that model checkpoints should be treated as a first-class data modality.<n>It argues that generative modeling in weight space should be standardized as a core machine learning primitive.<n>Our goal is to shift the community's default mindset from optimizing models per task to sampling models from learned weight distributions.
  arXiv  Detail & Related papers  (2026-05-18T16:38:26Z)
• Weight Space Representation Learning with Neural Fields [69.85677017502826]
  In this work, we investigate the potential of weights to serve as effective representations, focusing on neural fields.<n>Our key insight is that constraining the optimization space through a pre-trained base model and low-rank adaptation (LoRA) can induce structure in weight space.<n>Across reconstruction, generation, and analysis tasks on 2D and 3D data, we find that multiplicative LoRA weights achieve high representation quality while exhibiting distinctiveness and semantic structure.
  arXiv  Detail & Related papers  (2025-12-01T15:05:01Z)
• The Impact of Model Zoo Size and Composition on Weight Space Learning [8.11780615053558]
  Re-using trained neural network models is a common strategy to reduce training cost and transfer knowledge.<n>Weight space learning is a promising new field to re-use populations of pre-trained models for future tasks.<n>We propose a modification to a common weight space learning method to accommodate training on heterogeneous populations of models.
  arXiv  Detail & Related papers  (2025-04-14T11:54:06Z)
• Hyper-Representations: Learning from Populations of Neural Networks [3.8979646385036175]
  This thesis addresses the challenge of understanding Neural Networks through the lens of their most fundamental component: the weights. Work in this thesis finds that trained NN models indeed occupy meaningful structures in the weight space, that can be learned and used.
  arXiv  Detail & Related papers  (2024-10-07T15:03:00Z)
• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Improved Generalization of Weight Space Networks via Augmentations [53.87011906358727]
  Learning in deep weight spaces (DWS) is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs) We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces.
  arXiv  Detail & Related papers  (2024-02-06T15:34:44Z)
• Data Augmentations in Deep Weight Spaces [89.45272760013928]
  We introduce a novel augmentation scheme based on the Mixup method. We evaluate the performance of these techniques on existing benchmarks as well as new benchmarks we generate.
  arXiv  Detail & Related papers  (2023-11-15T10:43:13Z)
• Neural networks with late-phase weights [66.72777753269658]
  We show that the solutions found by SGD can be further improved by ensembling a subset of the weights in late stages of learning. At the end of learning, we obtain back a single model by taking a spatial average in weight space.
  arXiv  Detail & Related papers  (2020-07-25T13:23:37Z)
• Classifying the classifier: dissecting the weight space of neural networks [16.94879659770577]
  This paper presents an empirical study on the weights of neural networks. We interpret each model as a point in a high-dimensional space -- the neural weight space. To promote further research on the weight space, we release the neural weight space (NWS) dataset.
  arXiv  Detail & Related papers  (2020-02-13T18:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.