Classifying the classifier: dissecting the weight space of neural networks

• URL: http://arxiv.org/abs/2002.05688v1
• Date: Thu, 13 Feb 2020 18:12:02 GMT
• Title: Classifying the classifier: dissecting the weight space of neural networks
• Authors: Gabriel Eilertsen, Daniel J\"onsson, Timo Ropinski, Jonas Unger, Anders Ynnerman
• Abstract summary: This paper presents an empirical study on the weights of neural networks. We interpret each model as a point in a high-dimensional space -- the neural weight space. To promote further research on the weight space, we release the neural weight space (NWS) dataset.
• Score: 16.94879659770577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents an empirical study on the weights of neural networks, where we interpret each model as a point in a high-dimensional space -- the neural weight space. To explore the complex structure of this space, we sample from a diverse selection of training variations (dataset, optimization procedure, architecture, etc.) of neural network classifiers, and train a large number of models to represent the weight space. Then, we use a machine learning approach for analyzing and extracting information from this space. Most centrally, we train a number of novel deep meta-classifiers with the objective of classifying different properties of the training setup by identifying their footprints in the weight space. Thus, the meta-classifiers probe for patterns induced by hyper-parameters, so that we can quantify how much, where, and when these are encoded through the optimization process. This provides a novel and complementary view for explainable AI, and we show how meta-classifiers can reveal a great deal of information about the training setup and optimization, by only considering a small subset of randomly selected consecutive weights. To promote further research on the weight space, we release the neural weight space (NWS) dataset -- a collection of 320K weight snapshots from 16K individually trained deep neural networks.

Related papers

• Self Supervised Networks for Learning Latent Space Representations of Human Body Scans and Motions [6.165163123577484]
  This paper introduces self-supervised neural network models to tackle several fundamental problems in the field of 3D human body analysis and processing. We propose VariShaPE, a novel architecture for the retrieval of latent space representations of body shapes and poses. Second, we complement the estimation of latent codes with MoGeN, a framework that learns the geometry on the latent space itself.
  arXiv  Detail & Related papers  (2024-11-05T19:59:40Z)
• Neural Metamorphosis [72.88137795439407]
  This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta), which aims to build self-morphable neural networks. NeuMeta directly learns the continuous weight manifold of neural networks. It sustains full-size performance even at a 75% compression rate.
  arXiv  Detail & Related papers  (2024-10-10T14:49:58Z)
• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Improved Generalization of Weight Space Networks via Augmentations [53.87011906358727]
  Learning in deep weight spaces (DWS) is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs) We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces.
  arXiv  Detail & Related papers  (2024-02-06T15:34:44Z)
• Data Augmentations in Deep Weight Spaces [89.45272760013928]
  We introduce a novel augmentation scheme based on the Mixup method. We evaluate the performance of these techniques on existing benchmarks as well as new benchmarks we generate.
  arXiv  Detail & Related papers  (2023-11-15T10:43:13Z)
• Neural Functional Transformers [99.98750156515437]
  This paper uses the attention mechanism to define a novel set of permutation equivariant weight-space layers called neural functional Transformers (NFTs) NFTs respect weight-space permutation symmetries while incorporating the advantages of attention, which have exhibited remarkable success across multiple domains. We also leverage NFTs to develop Inr2Array, a novel method for computing permutation invariant representations from the weights of implicit neural representations (INRs)
  arXiv  Detail & Related papers  (2023-05-22T23:38:27Z)
• Transfer-Once-For-All: AI Model Optimization for Edge [0.0]
  We propose Transfer-Once-For-All (TOFA) for supernet-style training on small data sets with constant computational training cost. To overcome the challenges arising from small data, TOFA utilizes a unified semi-supervised training loss to simultaneously train all existings within the supernet.
  arXiv  Detail & Related papers  (2023-03-27T04:14:30Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• MLDS: A Dataset for Weight-Space Analysis of Neural Networks [0.0]
  We present MLDS, a new dataset consisting of thousands of trained neural networks with carefully controlled parameters. This dataset enables new insights into both model-to-model and model-to-training-data relationships.
  arXiv  Detail & Related papers  (2021-04-21T14:24:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.