Gradient Boosting Neural Networks: GrowNet

• URL: http://arxiv.org/abs/2002.07971v2
• Date: Sun, 14 Jun 2020 22:07:54 GMT
• Title: Gradient Boosting Neural Networks: GrowNet
• Authors: Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, Khoa Doan, and Sathiya S. Keerthi
• Abstract summary: A novel gradient boosting framework is proposed where shallow neural networks are employed as weak learners'' A fully corrective step is incorporated to remedy the pitfall of greedy function approximation of classic gradient boosting decision tree. The proposed model rendered outperforming results against state-of-the-art boosting methods in all three tasks on multiple datasets.
• Score: 9.0491536808974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A novel gradient boosting framework is proposed where shallow neural networks are employed as ``weak learners''. General loss functions are considered under this unified framework with specific examples presented for classification, regression, and learning to rank. A fully corrective step is incorporated to remedy the pitfall of greedy function approximation of classic gradient boosting decision tree. The proposed model rendered outperforming results against state-of-the-art boosting methods in all three tasks on multiple datasets. An ablation study is performed to shed light on the effect of each model components and model hyperparameters.

Related papers

• Theoretical Characterization of How Neural Network Pruning Affects its Generalization [131.1347309639727]
  This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization. It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero. More surprisingly, the generalization bound gets better as the pruning fraction gets larger.
  arXiv  Detail & Related papers  (2023-01-01T03:10:45Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Deep Manifold Learning with Graph Mining [80.84145791017968]
  We propose a novel graph deep model with a non-gradient decision layer for graph mining. The proposed model has achieved state-of-the-art performance compared to the current models.
  arXiv  Detail & Related papers  (2022-07-18T04:34:08Z)
• Scaling Private Deep Learning with Low-Rank and Sparse Gradients [5.14780936727027]
  We propose a framework that exploits the low-rank and sparse structure of neural networks to reduce the dimension of gradient updates. A novel strategy is utilized to sparsify the gradients, resulting in low-dimensional, less noisy updates. Empirical evaluation on natural language processing and computer vision tasks shows that our method outperforms other state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-07-06T14:09:47Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent [95.94432031144716]
  We propose a unified non- optimization framework for the analysis of a learning network. We show that existing guarantees can be trained unified through gradient descent.
  arXiv  Detail & Related papers  (2021-06-25T17:45:00Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• LaplaceNet: A Hybrid Energy-Neural Model for Deep Semi-Supervised Classification [0.0]
  Recent developments in deep semi-supervised classification have reached unprecedented performance. We propose a new framework, LaplaceNet, for deep semi-supervised classification that has a greatly reduced model complexity. Our model outperforms state-of-the-art methods for deep semi-supervised classification, over several benchmark datasets.
  arXiv  Detail & Related papers  (2021-06-08T17:09:28Z)
• Uncertainty in Gradient Boosting via Ensembles [37.808845398471874]
  ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. We propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.
  arXiv  Detail & Related papers  (2020-06-18T14:11:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.