Generative Feature Training of Thin 2-Layer Networks

• URL: http://arxiv.org/abs/2411.06848v1
• Date: Mon, 11 Nov 2024 10:32:33 GMT
• Title: Generative Feature Training of Thin 2-Layer Networks
• Authors: Johannes Hertrich, Sebastian Neumayer,
• Abstract summary: We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on squared loss and small datasets. As a highly hidden model, we exploit hidden weights with samples from learned distribution proposal. We refine the sampled weights with gradient-based post-processing in the latent space.
• Score: 0.0
• License:
• Abstract: We consider the approximation of functions by 2-layer neural networks with a small number of hidden weights based on the squared loss and small datasets. Due to the highly non-convex energy landscape, gradient-based training often suffers from local minima. As a remedy, we initialize the hidden weights with samples from a learned proposal distribution, which we parameterize as a deep generative model. To train this model, we exploit the fact that with fixed hidden weights, the optimal output weights solve a linear equation. After learning the generative model, we refine the sampled weights with a gradient-based post-processing in the latent space. Here, we also include a regularization scheme to counteract potential noise. Finally, we demonstrate the effectiveness of our approach by numerical examples.

Related papers

• Neural Metamorphosis [72.88137795439407]
  This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta), which aims to build self-morphable neural networks. NeuMeta directly learns the continuous weight manifold of neural networks. It sustains full-size performance even at a 75% compression rate.
  arXiv  Detail & Related papers  (2024-10-10T14:49:58Z)
• Bayes-optimal learning of an extensive-width neural network from quadratically many samples [28.315491743569897]
  We consider the problem of learning a target function corresponding to a single hidden layer neural network. We consider the limit where the input dimension and the network width are proportionally large.
  arXiv  Detail & Related papers  (2024-08-07T12:41:56Z)
• Efficient Training with Denoised Neural Weights [65.14892033932895]
  This work takes a novel step towards building a weight generator to synthesize the neural weights for initialization. We use the image-to-image translation task with generative adversarial networks (GANs) as an example due to the ease of collecting model weights. By initializing the image translation model with the denoised weights predicted by our diffusion model, the training requires only 43.3 seconds.
  arXiv  Detail & Related papers  (2024-07-16T17:59:42Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks [73.29587731448345]
  This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. Second, to obtain low bit-width activations, existing works consider all channels equally.
  arXiv  Detail & Related papers  (2020-12-26T15:21:18Z)
• A Bayesian Perspective on Training Speed and Model Selection [51.15664724311443]
  We show that a measure of a model's training speed can be used to estimate its marginal likelihood. We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks. Our results suggest a promising new direction towards explaining why neural networks trained with gradient descent are biased towards functions that generalize well.
  arXiv  Detail & Related papers  (2020-10-27T17:56:14Z)
• Neural networks with late-phase weights [66.72777753269658]
  We show that the solutions found by SGD can be further improved by ensembling a subset of the weights in late stages of learning. At the end of learning, we obtain back a single model by taking a spatial average in weight space.
  arXiv  Detail & Related papers  (2020-07-25T13:23:37Z)
• Depth-2 Neural Networks Under a Data-Poisoning Attack [2.105564340986074]
  We study the possibility of defending against data-poisoning attacks while training a shallow neural network in a regression setup. In this work, we focus on doing supervised learning for a class of depth-2 finite-width neural networks.
  arXiv  Detail & Related papers  (2020-05-04T17:56:15Z)
• DeepFit: 3D Surface Fitting via Neural Network Weighted Least Squares [43.24287146191367]
  We propose a surface fitting method for unstructured 3D point clouds. This method, called DeepFit, incorporates a neural network to learn point-wise weights for weighted least squares surface fitting.
  arXiv  Detail & Related papers  (2020-03-23T09:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.