Learning to Compose SuperWeights for Neural Parameter Allocation Search
- URL: http://arxiv.org/abs/2312.01274v1
- Date: Sun, 3 Dec 2023 04:20:02 GMT
- Title: Learning to Compose SuperWeights for Neural Parameter Allocation Search
- Authors: Piotr Teterwak, Soren Nelson, Nikoli Dryden, Dina Bashkirova, Kate
Saenko, Bryan A. Plummer
- Abstract summary: We show that our approach can generate parameters for many network using the same set of weights.
This enables us to support tasks like efficient ensembling and anytime prediction.
- Score: 61.078949532440724
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Neural parameter allocation search (NPAS) automates parameter sharing by
obtaining weights for a network given an arbitrary, fixed parameter budget.
Prior work has two major drawbacks we aim to address. First, there is a
disconnect in the sharing pattern between the search and training steps, where
weights are warped for layers of different sizes during the search to measure
similarity, but not during training, resulting in reduced performance. To
address this, we generate layer weights by learning to compose sets of
SuperWeights, which represent a group of trainable parameters. These
SuperWeights are created to be large enough so they can be used to represent
any layer in the network, but small enough that they are computationally
efficient. The second drawback we address is the method of measuring similarity
between shared parameters. Whereas prior work compared the weights themselves,
we argue this does not take into account the amount of conflict between the
shared weights. Instead, we use gradient information to identify layers with
shared weights that wish to diverge from each other. We demonstrate that our
SuperWeight Networks consistently boost performance over the state-of-the-art
on the ImageNet and CIFAR datasets in the NPAS setting. We further show that
our approach can generate parameters for many network architectures using the
same set of weights. This enables us to support tasks like efficient ensembling
and anytime prediction, outperforming fully-parameterized ensembles with 17%
fewer parameters.
Related papers
- Neural Metamorphosis [72.88137795439407]
This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta), which aims to build self-morphable neural networks.
NeuMeta directly learns the continuous weight manifold of neural networks.
It sustains full-size performance even at a 75% compression rate.
arXiv Detail & Related papers (2024-10-10T14:49:58Z) - Post-Training Quantization for Re-parameterization via Coarse & Fine
Weight Splitting [13.270381125055275]
We propose a coarse & fine weight splitting (CFWS) method to reduce quantization error of weight.
We develop an improved KL metric to determine optimal quantization scales for activation.
For example, the quantized RepVGG-A1 model exhibits a mere 0.3% accuracy loss.
arXiv Detail & Related papers (2023-12-17T02:31:20Z) - MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters [19.358670728803336]
Most deep neural networks are trained under fixed network architectures and require retraining when the architecture changes.
To avoid this, one can grow from a small network by adding random weights over time to gradually achieve the target network size.
This naive approach falls short in practice as it brings too much noise to the growing process.
arXiv Detail & Related papers (2023-11-07T11:37:08Z) - Weight Compander: A Simple Weight Reparameterization for Regularization [5.744133015573047]
We introduce weight compander, a novel effective method to improve generalization of deep neural networks.
We show experimentally that using weight compander in addition to standard regularization methods improves the performance of neural networks.
arXiv Detail & Related papers (2023-06-29T14:52:04Z) - DeepCuts: Single-Shot Interpretability based Pruning for BERT [0.0]
We show that our scoring functions are able to assign more relevant task-based scores to the network parameters.
We also analyze our pruning masks and find them to be significantly different from the ones obtained using standard metrics.
arXiv Detail & Related papers (2022-12-27T07:21:41Z) - Prompt Tuning for Parameter-efficient Medical Image Segmentation [79.09285179181225]
We propose and investigate several contributions to achieve a parameter-efficient but effective adaptation for semantic segmentation on two medical imaging datasets.
We pre-train this architecture with a dedicated dense self-supervision scheme based on assignments to online generated prototypes.
We demonstrate that the resulting neural network model is able to attenuate the gap between fully fine-tuned and parameter-efficiently adapted models.
arXiv Detail & Related papers (2022-11-16T21:55:05Z) - Parameter-Efficient Sparsity for Large Language Models Fine-Tuning [63.321205487234074]
We propose a.
sparse-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training.
Experiments with diverse networks (i.e., BERT, RoBERTa and GPT-2) demonstrate PST performs on par or better than previous sparsity methods.
arXiv Detail & Related papers (2022-05-23T02:43:45Z) - Generalizing Few-Shot NAS with Gradient Matching [165.5690495295074]
One-Shot methods train one supernet to approximate the performance of every architecture in the search space via weight-sharing.
Few-Shot NAS reduces the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets.
It significantly outperforms its Few-Shot counterparts while surpassing previous comparable methods in terms of the accuracy of derived architectures.
arXiv Detail & Related papers (2022-03-29T03:06:16Z) - Hyperparameter Ensembles for Robustness and Uncertainty Quantification [32.56745402836596]
Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration.
The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient.
In this paper, we design ensembles not only over weights, but over hyper parameters to improve the state of the art in both settings.
arXiv Detail & Related papers (2020-06-24T09:14:31Z) - How to Train Your Super-Net: An Analysis of Training Heuristics in
Weight-Sharing NAS [64.50415611717057]
We show that some commonly-used baselines for super-net training negatively impact the correlation between super-net and stand-alone performance.
Our code and experiments set a strong and reproducible baseline that future works can build on.
arXiv Detail & Related papers (2020-03-09T17:34:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.