Noise Optimization for Artificial Neural Networks

• URL: http://arxiv.org/abs/2102.04450v1
• Date: Sat, 6 Feb 2021 08:30:20 GMT
• Title: Noise Optimization for Artificial Neural Networks
• Authors: Li Xiao, Zeliang Zhang, Yijie Peng
• Abstract summary: We propose a new technique to compute the pathwise gradient estimate with respect to the standard deviation of the Gaussian noise added to each neuron of the ANN. In numerical experiments, our proposed method can achieve significant performance improvement on robustness of several popular ANN structures.
• Score: 0.973490996330539
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adding noises to artificial neural network(ANN) has been shown to be able to improve robustness in previous work. In this work, we propose a new technique to compute the pathwise stochastic gradient estimate with respect to the standard deviation of the Gaussian noise added to each neuron of the ANN. By our proposed technique, the gradient estimate with respect to noise levels is a byproduct of the backpropagation algorithm for estimating gradient with respect to synaptic weights in ANN. Thus, the noise level for each neuron can be optimized simultaneously in the processing of training the synaptic weights at nearly no extra computational cost. In numerical experiments, our proposed method can achieve significant performance improvement on robustness of several popular ANN structures under both black box and white box attacks tested in various computer vision datasets.

Related papers

• A lifted Bregman strategy for training unfolded proximal neural network Gaussian denoisers [8.343594411714934]
  Unfolded proximal neural networks (PNNs) form a family of methods that combines deep learning and proximal optimization approaches. We propose a lifted training formulation based on Bregman distances for unfolded PNNs. We assess the behaviour of the proposed training approach for PNNs through numerical simulations on image denoising.
  arXiv  Detail & Related papers  (2024-08-16T13:41:34Z)
• Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution [0.0]
  We propose an alternative descent learning equation based on quantized optimization for non- objective functions. We demonstrate the effectiveness of the proposed on vanilla neural convolution neural(CNN) models and the architecture across various data sets.
  arXiv  Detail & Related papers  (2023-05-30T08:55:59Z)
• Neural information coding for efficient spike-based image denoising [0.5156484100374058]
  In this work we investigate Spiking Neural Networks (SNNs) for Gaussian denoising. We propose a formal analysis of the information conversion processing carried out by the Leaky Integrate and Fire (LIF) neurons. We compare its performance with the classical rate-coding mechanism. Our results show that SNNs with LIF neurons can provide competitive denoising performance but at a reduced computational cost.
  arXiv  Detail & Related papers  (2023-05-15T09:05:32Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• A Novel Noise Injection-based Training Scheme for Better Model Robustness [9.749718440407811]
  Noise injection-based method has been shown to be able to improve the robustness of artificial neural networks. In this work, we propose a novel noise injection-based training scheme for better model robustness. Experiment results show that our proposed method achieves a much better performance on adversarial robustness and slightly better performance on original accuracy.
  arXiv  Detail & Related papers  (2023-02-17T02:50:25Z)
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo [97.07453889070574]
  We present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors. We show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes.
  arXiv  Detail & Related papers  (2021-09-02T17:54:31Z)
• Learning Frequency Domain Approximation for Binary Neural Networks [68.79904499480025]
  We propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2021-03-01T08:25:26Z)
• Deep Networks for Direction-of-Arrival Estimation in Low SNR [89.45026632977456]
  We introduce a Convolutional Neural Network (CNN) that is trained from mutli-channel data of the true array manifold matrix. We train a CNN in the low-SNR regime to predict DoAs across all SNRs. Our robust solution can be applied in several fields, ranging from wireless array sensors to acoustic microphones or sonars.
  arXiv  Detail & Related papers  (2020-11-17T12:52:18Z)
• Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks [77.34726150561087]
  We introduce Gradient Markov Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.
  arXiv  Detail & Related papers  (2020-08-25T15:48:15Z)
• Evolving Deep Convolutional Neural Networks for Hyperspectral Image Denoising [6.869192200282213]
  We propose a novel algorithm to automatically build an optimal Convolutional Neural Network (CNN) to effectively denoise HSIs. The experiments of the proposed algorithm have been well-designed and compared against the state-of-the-art peer competitors.
  arXiv  Detail & Related papers  (2020-08-15T03:04:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.