Classifying the classifier: dissecting the weight space of neural
networks
- URL: http://arxiv.org/abs/2002.05688v1
- Date: Thu, 13 Feb 2020 18:12:02 GMT
- Title: Classifying the classifier: dissecting the weight space of neural
networks
- Authors: Gabriel Eilertsen, Daniel J\"onsson, Timo Ropinski, Jonas Unger,
Anders Ynnerman
- Abstract summary: This paper presents an empirical study on the weights of neural networks.
We interpret each model as a point in a high-dimensional space -- the neural weight space.
To promote further research on the weight space, we release the neural weight space (NWS) dataset.
- Score: 16.94879659770577
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents an empirical study on the weights of neural networks,
where we interpret each model as a point in a high-dimensional space -- the
neural weight space. To explore the complex structure of this space, we sample
from a diverse selection of training variations (dataset, optimization
procedure, architecture, etc.) of neural network classifiers, and train a large
number of models to represent the weight space. Then, we use a machine learning
approach for analyzing and extracting information from this space. Most
centrally, we train a number of novel deep meta-classifiers with the objective
of classifying different properties of the training setup by identifying their
footprints in the weight space. Thus, the meta-classifiers probe for patterns
induced by hyper-parameters, so that we can quantify how much, where, and when
these are encoded through the optimization process. This provides a novel and
complementary view for explainable AI, and we show how meta-classifiers can
reveal a great deal of information about the training setup and optimization,
by only considering a small subset of randomly selected consecutive weights. To
promote further research on the weight space, we release the neural weight
space (NWS) dataset -- a collection of 320K weight snapshots from 16K
individually trained deep neural networks.
Related papers
- Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
This paper introduces the SANE approach to weight-space learning.
Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
arXiv Detail & Related papers (2024-06-14T13:12:07Z) - Improved Generalization of Weight Space Networks via Augmentations [56.571475005291035]
Learning in deep weight spaces (DWS) is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs)
We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets.
To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces.
arXiv Detail & Related papers (2024-02-06T15:34:44Z) - Data Augmentations in Deep Weight Spaces [89.45272760013928]
We introduce a novel augmentation scheme based on the Mixup method.
We evaluate the performance of these techniques on existing benchmarks as well as new benchmarks we generate.
arXiv Detail & Related papers (2023-11-15T10:43:13Z) - Spiking mode-based neural networks [2.5690340428649328]
Spiking neural networks play an important role in brain-like neuromorphic computations and in studying working mechanisms of neural circuits.
One drawback of training a large scale spiking neural network is that updating all weights is quite expensive.
We propose a spiking mode-based training protocol, where the recurrent weight matrix is explained as a Hopfield-like multiplication of three matrices.
arXiv Detail & Related papers (2023-10-23T06:54:17Z) - Neural Functional Transformers [99.98750156515437]
This paper uses the attention mechanism to define a novel set of permutation equivariant weight-space layers called neural functional Transformers (NFTs)
NFTs respect weight-space permutation symmetries while incorporating the advantages of attention, which have exhibited remarkable success across multiple domains.
We also leverage NFTs to develop Inr2Array, a novel method for computing permutation invariant representations from the weights of implicit neural representations (INRs)
arXiv Detail & Related papers (2023-05-22T23:38:27Z) - Transfer-Once-For-All: AI Model Optimization for Edge [0.0]
We propose Transfer-Once-For-All (TOFA) for supernet-style training on small data sets with constant computational training cost.
To overcome the challenges arising from small data, TOFA utilizes a unified semi-supervised training loss to simultaneously train all existings within the supernet.
arXiv Detail & Related papers (2023-03-27T04:14:30Z) - Permutation Equivariant Neural Functionals [92.0667671999604]
This work studies the design of neural networks that can process the weights or gradients of other neural networks.
We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order.
In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
arXiv Detail & Related papers (2023-02-27T18:52:38Z) - MLDS: A Dataset for Weight-Space Analysis of Neural Networks [0.0]
We present MLDS, a new dataset consisting of thousands of trained neural networks with carefully controlled parameters.
This dataset enables new insights into both model-to-model and model-to-training-data relationships.
arXiv Detail & Related papers (2021-04-21T14:24:26Z) - Learning Neural Network Subspaces [74.44457651546728]
Recent observations have advanced our understanding of the neural network optimization landscape.
With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.
With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.
arXiv Detail & Related papers (2021-02-20T23:26:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.