FocusedDropout for Convolutional Neural Network

• URL: http://arxiv.org/abs/2103.15425v1
• Date: Mon, 29 Mar 2021 08:47:55 GMT
• Title: FocusedDropout for Convolutional Neural Network
• Authors: Tianshu Xie, Minghui Liu, Jiali Deng, Xuan Cheng, Xiaomin Wang, Ming Liu
• Abstract summary: FocusedDropout is a non-random dropout method to make the network focus more on the target. Even a slight cost, 10% of batches employing FocusedDropout, can produce a nice performance boost over the baselines.
• Score: 6.066543113636522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In convolutional neural network (CNN), dropout cannot work well because dropped information is not entirely obscured in convolutional layers where features are correlated spatially. Except randomly discarding regions or channels, many approaches try to overcome this defect by dropping influential units. In this paper, we propose a non-random dropout method named FocusedDropout, aiming to make the network focus more on the target. In FocusedDropout, we use a simple but effective way to search for the target-related features, retain these features and discard others, which is contrary to the existing methods. We found that this novel method can improve network performance by making the network more target-focused. Besides, increasing the weight decay while using FocusedDropout can avoid the overfitting and increase accuracy. Experimental results show that even a slight cost, 10\% of batches employing FocusedDropout, can produce a nice performance boost over the baselines on multiple datasets of classification, including CIFAR10, CIFAR100, Tiny Imagenet, and has a good versatility for different CNN models.

Related papers

• Towards Meta-Pruning via Optimal Transport [64.6060250923073]
  This paper introduces a novel approach named Intra-Fusion, challenging the prevailing pruning paradigm. We leverage the concepts of model fusion and Optimal Transport to arrive at a more effective sparse model representation. We benchmark our results for various networks on commonly used datasets such as CIFAR-10, CIFAR-100, and ImageNet.
  arXiv  Detail & Related papers  (2024-02-12T17:50:56Z)
• Neuron-Specific Dropout: A Deterministic Regularization Technique to Prevent Neural Networks from Overfitting & Reduce Dependence on Large Training Samples [0.0]
  NSDropout looks at both the training pass, and validation pass, of a layer in a model. By comparing the average values produced by each neuron for each class in a data set, the network is able to drop targeted units. The layer is able to predict what features, or noise, the model is looking at during testing that isn't present when looking at samples from validation.
  arXiv  Detail & Related papers  (2022-01-13T13:10:30Z)
• BreakingBED -- Breaking Binary and Efficient Deep Neural Networks by Adversarial Attacks [65.2021953284622]
  We study robustness of CNNs against white-box and black-box adversarial attacks. Results are shown for distilled CNNs, agent-based state-of-the-art pruned models, and binarized neural networks.
  arXiv  Detail & Related papers  (2021-03-14T20:43:19Z)
• AutoDropout: Learning Dropout Patterns to Regularize Deep Networks [82.28118615561912]
  Dropout or weight decay methods do not leverage the structures of the network's inputs and hidden states. We show that this method works well for both image recognition on CIFAR-10 and ImageNet, as well as language modeling on Penn Treebank and WikiText-2. The learned dropout patterns also transfers to different tasks and datasets, such as from language model on Penn Treebank to Engligh-French translation on WMT 2014.
  arXiv  Detail & Related papers  (2021-01-05T19:54:22Z)
• SelectScale: Mining More Patterns from Images via Selective and Soft Dropout [35.066419181817594]
  Convolutional neural networks (CNNs) have achieved remarkable success in image recognition. We propose SelectScale, which selects the important features in networks and adjusts them during training. Using SelectScale, we improve the performance of CNNs on CIFAR and ImageNet.
  arXiv  Detail & Related papers  (2020-11-30T12:15:08Z)
• TargetDrop: A Targeted Regularization Method for Convolutional Neural Networks [6.014015535168499]
  Dropout regularization has been widely used in deep learning but performs less effective for convolutional neural networks. We propose a targeted regularization method named TargetDrop which incorporates the attention mechanism to drop the discriminative feature units.
  arXiv  Detail & Related papers  (2020-10-21T02:26:05Z)
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization [62.8384110757689]
  Overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs) The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets.
  arXiv  Detail & Related papers  (2020-10-11T13:19:58Z)
• Informative Dropout for Robust Representation Learning: A Shape-bias Perspective [84.30946377024297]
  We propose a light-weight model-agnostic method, namely Informative Dropout (InfoDrop), to improve interpretability and reduce texture bias. Specifically, we discriminate texture from shape based on local self-information in an image, and adopt a Dropout-like algorithm to decorrelate the model output from the local texture.
  arXiv  Detail & Related papers  (2020-08-10T16:52:24Z)
• MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values [0.0]
  MaxDropout is a regularizer for deep neural network models that works in a supervised fashion by removing prominent neurons. We show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout.
  arXiv  Detail & Related papers  (2020-07-27T17:55:54Z)
• EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks [45.4796383952516]
  We propose EDropout as an energy-based framework for pruning neural networks in classification tasks. A set of binary pruning state vectors (population) represents a set of corresponding sub-networks from an arbitrary provided original neural network. EDropout can prune typical neural networks without modification of the network architecture.
  arXiv  Detail & Related papers  (2020-06-07T21:09:44Z)
• Cooling-Shrinking Attack: Blinding the Tracker with Imperceptible Noises [87.53808756910452]
  A cooling-shrinking attack method is proposed to deceive state-of-the-art SiameseRPN-based trackers. Our method has good transferability and is able to deceive other top-performance trackers such as DaSiamRPN, DaSiamRPN-UpdateNet, and DiMP.
  arXiv  Detail & Related papers  (2020-03-21T07:13:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.