To Boost or not to Boost: On the Limits of Boosted Neural Networks

• URL: http://arxiv.org/abs/2107.13600v1
• Date: Wed, 28 Jul 2021 19:10:03 GMT
• Title: To Boost or not to Boost: On the Limits of Boosted Neural Networks
• Authors: Sai Saketh Rambhatla, Michael Jones, Rama Chellappa
• Abstract summary: Boosting is a method for learning an ensemble of classifiers. While boosting has been shown to be very effective for decision trees, its impact on neural networks has not been extensively studied. We find that a single neural network usually generalizes better than a boosted ensemble of smaller neural networks with the same total number of parameters.
• Score: 67.67776094785363
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Boosting is a method for finding a highly accurate hypothesis by linearly combining many ``weak" hypotheses, each of which may be only moderately accurate. Thus, boosting is a method for learning an ensemble of classifiers. While boosting has been shown to be very effective for decision trees, its impact on neural networks has not been extensively studied. We prove one important difference between sums of decision trees compared to sums of convolutional neural networks (CNNs) which is that a sum of decision trees cannot be represented by a single decision tree with the same number of parameters while a sum of CNNs can be represented by a single CNN. Next, using standard object recognition datasets, we verify experimentally the well-known result that a boosted ensemble of decision trees usually generalizes much better on testing data than a single decision tree with the same number of parameters. In contrast, using the same datasets and boosting algorithms, our experiments show the opposite to be true when using neural networks (both CNNs and multilayer perceptrons (MLPs)). We find that a single neural network usually generalizes better than a boosted ensemble of smaller neural networks with the same total number of parameters.

Related papers

• RF-GNN: Random Forest Boosted Graph Neural Network for Social Bot Detection [10.690802468726078]
  The presence of a large number of bots on social media leads to adverse effects. This paper proposes a Random Forest boosted Graph Neural Network for social bot detection, called RF-GNN.
  arXiv  Detail & Related papers  (2023-04-14T00:57:44Z)
• Structured Bayesian Compression for Deep Neural Networks Based on The Turbo-VBI Approach [23.729955669774977]
  In most existing pruning methods, surviving neurons are randomly connected in the neural network without any structure. We propose a three-layer hierarchical prior to promote a more regular sparse structure during pruning. We derive an efficient Turbo-variational Bayesian inferencing (Turbo-VBI) algorithm to solve the resulting model compression problem.
  arXiv  Detail & Related papers  (2023-02-21T07:12:36Z)
• Improving the Accuracy and Robustness of CNNs Using a Deep CCA Neural Data Regularizer [2.026424957803652]
  As convolutional neural networks (CNNs) become more accurate at object recognition, their representations become more similar to the primate visual system. Previous attempts to address this question showed very modest gains in accuracy, owing in part to limitations of the regularization method. We develop a new neural data regularizer for CNNs that uses Deep Correlation Analysis (DCCA) to optimize the resemblance of the CNN's image representations to that of the monkey visual cortex.
  arXiv  Detail & Related papers  (2022-09-06T15:40:39Z)
• Coin Flipping Neural Networks [8.009932864430901]
  We show that neural networks with access to randomness can outperform deterministic networks by using amplification. We conjecture that for most classification problems, there is a CFNN which solves them with higher accuracy or fewer neurons than any deterministic network.
  arXiv  Detail & Related papers  (2022-06-18T11:19:44Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• Distilled Neural Networks for Efficient Learning to Rank [0.0]
  We propose an approach for speeding up neural scoring time by applying a combination of Distillation, Pruning and Fast Matrix multiplication. Comprehensive experiments on two public learning-to-rank datasets show that neural networks produced with our novel approach are competitive at any point of the effectiveness-efficiency trade-off.
  arXiv  Detail & Related papers  (2022-02-22T08:40:18Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• Growing Deep Forests Efficiently with Soft Routing and Learned Connectivity [79.83903179393164]
  This paper further extends the deep forest idea in several important aspects. We employ a probabilistic tree whose nodes make probabilistic routing decisions, a.k.a., soft routing, rather than hard binary decisions. Experiments on the MNIST dataset demonstrate that our empowered deep forests can achieve better or comparable performance than [1],[3].
  arXiv  Detail & Related papers  (2020-12-29T18:05:05Z)
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution [57.635467829558664]
  We introduce a structural regularization across convolutional kernels in a CNN. We show that CNNs now maintain performance with dramatic reduction in parameters and computations.
  arXiv  Detail & Related papers  (2020-09-04T20:41:47Z)
• Neural Additive Models: Interpretable Machine Learning with Neural Nets [77.66871378302774]
  Deep neural networks (DNNs) are powerful black-box predictors that have achieved impressive performance on a wide variety of tasks. We propose Neural Additive Models (NAMs) which combine some of the expressivity of DNNs with the inherent intelligibility of generalized additive models. NAMs learn a linear combination of neural networks that each attend to a single input feature.
  arXiv  Detail & Related papers  (2020-04-29T01:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.