Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks

• URL: http://arxiv.org/abs/2208.07265v1
• Date: Mon, 15 Aug 2022 15:17:34 GMT
• Title: Combining Gradients and Probabilities for Heterogeneous Approximation of Neural Networks
• Authors: Elias Trommer, Bernd Waschneck, Akash Kumar
• Abstract summary: We discuss the validity of additive Gaussian noise as a surrogate model for behavioral simulation of approximate multipliers. The amount of noise injected into the accurate computations is learned during network training using backpropagation. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for variants.
• Score: 2.5744053804694893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work explores the search for heterogeneous approximate multiplier configurations for neural networks that produce high accuracy and low energy consumption. We discuss the validity of additive Gaussian noise added to accurate neural network computations as a surrogate model for behavioral simulation of approximate multipliers. The continuous and differentiable properties of the solution space spanned by the additive Gaussian noise model are used as a heuristic that generates meaningful estimates of layer robustness without the need for combinatorial optimization techniques. Instead, the amount of noise injected into the accurate computations is learned during network training using backpropagation. A probabilistic model of the multiplier error is presented to bridge the gap between the domains; the model estimates the standard deviation of the approximate multiplier error, connecting solutions in the additive Gaussian noise space to actual hardware instances. Our experiments show that the combination of heterogeneous approximation and neural network retraining reduces the energy consumption for multiplications by 70% to 79% for different ResNet variants on the CIFAR-10 dataset with a Top-1 accuracy loss below one percentage point. For the more complex Tiny ImageNet task, our VGG16 model achieves a 53 % reduction in energy consumption with a drop in Top-5 accuracy of 0.5 percentage points. We further demonstrate that our error model can predict the parameters of an approximate multiplier in the context of the commonly used additive Gaussian noise (AGN) model with high accuracy. Our software implementation is available under https://github.com/etrommer/agn-approx.

Related papers

• Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection [11.729744197698718]
  We present an algorithmic framework to approximate a neural network of finite width and depth. We iteratively approximate the output distribution of each layer of the neural network as a mixture of Gaussian processes. Our results can represent an important step towards understanding neural network predictions.
  arXiv  Detail & Related papers  (2024-07-26T12:45:53Z)
• Towards Generalized Entropic Sparsification for Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are reported to be overparametrized. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint.
  arXiv  Detail & Related papers  (2024-04-06T21:33:39Z)
• Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes [23.87733307119697]
  We introduce Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a sampler and minimizes the Regularized Stein Discrepancy in L2 space between the generated distribution and true posterior. We demonstrate that the bias introduced by our method can be controlled by multiplying the divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm.
  arXiv  Detail & Related papers  (2023-09-22T06:56:35Z)
• Geometry-Informed Neural Operator for Large-Scale 3D PDEs [76.06115572844882]
  We propose the geometry-informed neural operator (GINO) to learn the solution operator of large-scale partial differential equations. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points.
  arXiv  Detail & Related papers  (2023-09-01T16:59:21Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• Automatic Network Adaptation for Ultra-Low Uniform-Precision Quantization [6.1664476076961146]
  Uniform-precision neural network quantization has gained popularity since it simplifies densely packed arithmetic unit for high computing capability. It ignores heterogeneous sensitivity to the impact of quantization errors across the layers, resulting in sub-optimal inference. This work proposes a novel neural architecture search called neural channel expansion that adjusts the network structure to alleviate accuracy degradation from ultra-low uniform-precision quantization.
  arXiv  Detail & Related papers  (2022-12-21T09:41:25Z)
• Fast Exploration of the Impact of Precision Reduction on Spiking Neural Networks [63.614519238823206]
  Spiking Neural Networks (SNNs) are a practical choice when the target hardware reaches the edge of computing. We employ an Interval Arithmetic (IA) model to develop an exploration methodology that takes advantage of the capability of such a model to propagate the approximation error.
  arXiv  Detail & Related papers  (2022-11-22T15:08:05Z)
• SignalNet: A Low Resolution Sinusoid Decomposition and Estimation Network [79.04274563889548]
  We propose SignalNet, a neural network architecture that detects the number of sinusoids and estimates their parameters from quantized in-phase and quadrature samples. We introduce a worst-case learning threshold for comparing the results of our network relative to the underlying data distributions. In simulation, we find that our algorithm is always able to surpass the threshold for three-bit data but often cannot exceed the threshold for one-bit data.
  arXiv  Detail & Related papers  (2021-06-10T04:21:20Z)
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models [56.21470608621633]
  We propose a time estimation framework to decouple the architectural search from the target hardware. The proposed methodology extracts a set of models from micro- kernel and multi-layer benchmarks and generates a stacked model for mapping and network execution time estimation. We compare estimation accuracy and fidelity of the generated mixed models, statistical models with the roofline model, and a refined roofline model for evaluation.
  arXiv  Detail & Related papers  (2021-05-07T11:39:05Z)
• A Bayesian regularization-backpropagation neural network model for peeling computations [0.0]
  The input data is taken from finite element (FE) peeling results. The neural network is trained with 75% of the FE dataset. It is shown that the BR-BPNN model in conjunction with k-fold technique has significant potential to estimate the peeling behavior.
  arXiv  Detail & Related papers  (2020-06-29T21:58:43Z)
• Neural Control Variates [71.42768823631918]
  We show that a set of neural networks can face the challenge of finding a good approximation of the integrand. We derive a theoretically optimal, variance-minimizing loss function, and propose an alternative, composite loss for stable online training in practice. Specifically, we show that the learned light-field approximation is of sufficient quality for high-order bounces, allowing us to omit the error correction and thereby dramatically reduce the noise at the cost of negligible visible bias.
  arXiv  Detail & Related papers  (2020-06-02T11:17:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.