Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network

• URL: http://arxiv.org/abs/2106.07026v1
• Date: Sun, 13 Jun 2021 15:33:49 GMT
• Title: Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network
• Authors: Zhicheng Cai, Kaizhu Huang, Chenglei Peng
• Abstract summary: This paper proposes a novel nonlinear activation mechanism typically for convolutional neural network (CNN) In sharp contrast to ReLU which cuts off the negative phase value, the reborn mechanism enjoys the capacity to reconstruct dead neurons.
• Score: 14.929863072047318
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a novel nonlinear activation mechanism typically for convolutional neural network (CNN), named as reborn mechanism. In sharp contrast to ReLU which cuts off the negative phase value, the reborn mechanism enjoys the capacity to reborn and reconstruct dead neurons. Compared to other improved ReLU functions, reborn mechanism introduces a more proper way to utilize the negative phase information. Extensive experiments validate that this activation mechanism is able to enhance the model representation ability more significantly and make the better use of the input data information while maintaining the advantages of the original ReLU function. Moreover, reborn mechanism enables a non-symmetry that is hardly achieved by traditional CNNs and can act as a channel compensation method, offering competitive or even better performance but with fewer learned parameters than traditional methods. Reborn mechanism was tested on various benchmark datasets, all obtaining better performance than previous nonlinear activation functions.

Related papers

• Self-Normalized Resets for Plasticity in Continual Learning [6.5990719141691825]
  Self-Normalized Resets (SNR) is an adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero. We show that SNR consistently attains superior performance compared to its competitor algorithms.
  arXiv  Detail & Related papers  (2024-10-26T06:47:13Z)
• Feedback Favors the Generalization of Neural ODEs [24.342023073252395]
  We present feedback neural networks, showing that a feedback loop can flexibly correct the learned latent dynamics of neural ordinary differential equations (neural ODEs) The feedback neural network is a novel two-DOF neural network, which possesses robust performance in unseen scenarios with no loss of accuracy performance on previous tasks.
  arXiv  Detail & Related papers  (2024-10-14T08:09:45Z)
• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• Spiking Neural Networks with Consistent Mapping Relations Allow High-Accuracy Inference [9.667807887916132]
  Spike-based neuromorphic hardware has demonstrated substantial potential in low energy consumption and efficient inference. Direct training of deep spiking neural networks is challenging, and conversion-based methods still require substantial time delay owing to unresolved conversion errors.
  arXiv  Detail & Related papers  (2024-06-08T06:40:00Z)
• Learn to Enhance the Negative Information in Convolutional Neural Network [6.910916428810853]
  This paper proposes a learnable nonlinear activation mechanism specifically for convolutional neural network (CNN) termed as LENI. In sharp contrast to ReLU which cuts off the negative neurons and suffers from the issue of ''dying ReLU'', LENI enjoys the capacity to reconstruct the dead neurons and reduce the information loss.
  arXiv  Detail & Related papers  (2023-06-18T12:02:36Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization [9.34612743192798]
  Spiking neural networks are a promising approach towards next-generation models of the brain in computational neuroscience. We apply end-to-end learning with membrane potential-based backpropagation to a spiking convolutional auto-encoder. We show that applying regularization on membrane potential and spiking output successfully avoids both dead and bursting neurons.
  arXiv  Detail & Related papers  (2021-09-22T21:27:40Z)
• Relaxing the Constraints on Predictive Coding Models [62.997667081978825]
  Predictive coding is an influential theory of cortical function which posits that the principal computation the brain performs is the minimization of prediction errors. Standard implementations of the algorithm still involve potentially neurally implausible features such as identical forward and backward weights, backward nonlinear derivatives, and 1-1 error unit connectivity. In this paper, we show that these features are not integral to the algorithm and can be removed either directly or through learning additional sets of parameters with Hebbian update rules without noticeable harm to learning performance.
  arXiv  Detail & Related papers  (2020-10-02T15:21:37Z)
• Lipschitz Recurrent Neural Networks [100.72827570987992]
  We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
  arXiv  Detail & Related papers  (2020-06-22T08:44:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.