MaxDropoutV2: An Improved Method to Drop out Neurons in Convolutional Neural Networks

• URL: http://arxiv.org/abs/2203.02740v1
• Date: Sat, 5 Mar 2022 13:41:56 GMT
• Title: MaxDropoutV2: An Improved Method to Drop out Neurons in Convolutional Neural Networks
• Authors: Claudio Filipi Goncalves do Santos, Mateus Roder, Leandro A. Passos, and Jo\~ao P. Papa
• Abstract summary: We present an improved version of a supervised regularization technique called MaxDropoutV2. Results show that the model performs faster than the standard version and, in most cases, provides more accurate results.
• Score: 0.39146761527401425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the last decade, exponential data growth supplied the machine learning-based algorithms' capacity and enabled their usage in daily life activities. Additionally, such an improvement is partially explained due to the advent of deep learning techniques, i.e., stacks of simple architectures that end up in more complex models. Although both factors produce outstanding results, they also pose drawbacks regarding the learning process since training complex models denotes an expensive task and results are prone to overfit the training data. A supervised regularization technique called MaxDropout was recently proposed to tackle the latter, providing several improvements concerning traditional regularization approaches. In this paper, we present its improved version called MaxDropoutV2. Results considering two public datasets show that the model performs faster than the standard version and, in most cases, provides more accurate results.

Related papers

• Accelerating Deep Learning with Fixed Time Budget [2.190627491782159]
  This paper proposes an effective technique for training arbitrary deep learning models within fixed time constraints. The proposed method is extensively evaluated in both classification and regression tasks in computer vision.
  arXiv  Detail & Related papers  (2024-10-03T21:18:04Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Optimizing Dense Feed-Forward Neural Networks [0.0]
  We propose a novel feed-forward neural network constructing method based on pruning and transfer learning. Our approach can compress the number of parameters by more than 70%. We also evaluate the transfer learning level comparing the refined model and the original one training from scratch a neural network.
  arXiv  Detail & Related papers  (2023-12-16T23:23:16Z)
• The Trifecta: Three simple techniques for training deeper Forward-Forward networks [0.0]
  We propose a collection of three techniques that synergize exceptionally well and drastically improve the Forward-Forward algorithm on deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets.
  arXiv  Detail & Related papers  (2023-11-29T22:44:32Z)
• Robust Learning with Progressive Data Expansion Against Spurious Correlation [65.83104529677234]
  We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features. Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process. We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
  arXiv  Detail & Related papers  (2023-06-08T05:44:06Z)
• Towards Compute-Optimal Transfer Learning [82.88829463290041]
  We argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.
  arXiv  Detail & Related papers  (2023-04-25T21:49:09Z)
• PIVOT: Prompting for Video Continual Learning [50.80141083993668]
  We introduce PIVOT, a novel method that leverages extensive knowledge in pre-trained models from the image domain. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.
  arXiv  Detail & Related papers  (2022-12-09T13:22:27Z)
• From Actions to Events: A Transfer Learning Approach Using Improved Deep Belief Networks [1.0554048699217669]
  This paper proposes a novel approach to map the knowledge from action recognition to event recognition using an energy-based model. Such a model can process all frames simultaneously, carrying spatial and temporal information through the learning process.
  arXiv  Detail & Related papers  (2022-11-30T14:47:10Z)
• Regularizing Generative Adversarial Networks under Limited Data [88.57330330305535]
  This work proposes a regularization approach for training robust GAN models on limited data. We show a connection between the regularized loss and an f-divergence called LeCam-divergence, which we find is more robust under limited training data.
  arXiv  Detail & Related papers  (2021-04-07T17:59:06Z)
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization [62.8384110757689]
  Overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs) The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets.
  arXiv  Detail & Related papers  (2020-10-11T13:19:58Z)
• An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillation [1.433758865948252]
  We propose a new formalism of knowledge distillation for regression problems. First, we propose a new loss function, teacher outlier loss rejection, which rejects outliers in training samples using teacher model predictions. By considering the multi-task network, training of the feature extraction of student models becomes more effective.
  arXiv  Detail & Related papers  (2020-02-28T08:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.