Dynamic DropConnect: Enhancing Neural Network Robustness through Adaptive Edge Dropping Strategies

• URL: http://arxiv.org/abs/2502.19948v1
• Date: Thu, 27 Feb 2025 10:17:02 GMT
• Title: Dynamic DropConnect: Enhancing Neural Network Robustness through Adaptive Edge Dropping Strategies
• Authors: Yuan-Chih Yang, Hung-Hsuan Chen,
• Abstract summary: Dropout and DropConnect are well-known techniques that apply a consistent drop rate to randomly deactivate neurons or edges in a neural network layer during training.<n>This paper introduces a novel methodology that assigns dynamic drop rates to each edge within a layer, uniquely tailoring the dropping process without incorporating additional learning parameters.
• Score: 2.07180164747172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dropout and DropConnect are well-known techniques that apply a consistent drop rate to randomly deactivate neurons or edges in a neural network layer during training. This paper introduces a novel methodology that assigns dynamic drop rates to each edge within a layer, uniquely tailoring the dropping process without incorporating additional learning parameters. We perform experiments on synthetic and openly available datasets to validate the effectiveness of our approach. The results demonstrate that our method outperforms Dropout, DropConnect, and Standout, a classic mechanism known for its adaptive dropout capabilities. Furthermore, our approach improves the robustness and generalization of neural network training without increasing computational complexity. The complete implementation of our methodology is publicly accessible for research and replication purposes at https://github.com/ericabd888/Adjusting-the-drop-probability-in-DropConnect-based-on-the-magnitude-o f-the-gradient/.

Related papers

• Sharpening Your Density Fields: Spiking Neuron Aided Fast Geometry Learning [8.657209169726977]
  We introduce a spiking neuron mechanism that dynamically adjusts the threshold, eliminating the need for manual selection.<n>We validate our approach through extensive experiments on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-12-13T05:51:03Z)
• Y-Drop: A Conductance based Dropout for fully connected layers [63.029110722758496]
  We introduce Y-Drop, a regularization method that biases the dropout algorithm towards dropping more important neurons with higher probability. We show that forcing the network to solve the task at hand in the absence of its important units yields a strong regularization effect.
  arXiv  Detail & Related papers  (2024-09-11T15:56:08Z)
• Adaptive Class Emergence Training: Enhancing Neural Network Stability and Generalization through Progressive Target Evolution [0.0]
  We propose a novel training methodology for neural networks in classification problems. We evolve the target outputs from a null vector to one-hot encoded vectors throughout the training process. This gradual transition allows the network to adapt more smoothly to the increasing complexity of the classification task.
  arXiv  Detail & Related papers  (2024-09-04T03:25:48Z)
• Robust Neural Pruning with Gradient Sampling Optimization for Residual Neural Networks [0.0]
  This research embarks on pioneering the integration of gradient sampling optimization techniques, particularly StochGradAdam, into the pruning process of neural networks. Our main objective is to address the significant challenge of maintaining accuracy in pruned neural models, critical in resource-constrained scenarios.
  arXiv  Detail & Related papers  (2023-12-26T12:19:22Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization [62.8384110757689]
  Overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs) The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets.
  arXiv  Detail & Related papers  (2020-10-11T13:19:58Z)
• MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values [0.0]
  MaxDropout is a regularizer for deep neural network models that works in a supervised fashion by removing prominent neurons. We show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout.
  arXiv  Detail & Related papers  (2020-07-27T17:55:54Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Exploiting Non-Linear Redundancy for Neural Model Compression [26.211513643079993]
  We propose a novel model compression approach based on exploitation of linear dependence. Our method results in a reduction of up to 99% in overall network size with small loss in performance.
  arXiv  Detail & Related papers  (2020-05-28T15:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.