Residual Likelihood Forests
- URL: http://arxiv.org/abs/2011.02086v1
- Date: Wed, 4 Nov 2020 00:59:41 GMT
- Title: Residual Likelihood Forests
- Authors: Yan Zuo, Tom Drummond
- Abstract summary: This paper presents a novel ensemble learning approach called Residual Likelihood Forests (RLF)
Our weak learners produce conditional likelihoods that are sequentially optimized using global loss in the context of previous learners.
When compared against several ensemble approaches including Random Forests and Gradient Boosted Trees, RLFs offer a significant improvement in performance.
- Score: 19.97069303172077
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents a novel ensemble learning approach called Residual
Likelihood Forests (RLF). Our weak learners produce conditional likelihoods
that are sequentially optimized using global loss in the context of previous
learners within a boosting-like framework (rather than probability
distributions that are measured from observed data) and are combined
multiplicatively (rather than additively). This increases the efficiency of our
strong classifier, allowing for the design of classifiers which are more
compact in terms of model capacity. We apply our method to several machine
learning classification tasks, showing significant improvements in performance.
When compared against several ensemble approaches including Random Forests and
Gradient Boosted Trees, RLFs offer a significant improvement in performance
whilst concurrently reducing the required model size.
Related papers
- Extension of Transformational Machine Learning: Classification Problems [0.0]
This study explores the application and performance of Transformational Machine Learning (TML) in drug discovery.
TML, a meta learning algorithm, excels in exploiting common attributes across various domains.
The drug discovery process, which is complex and time-consuming, can benefit greatly from the enhanced prediction accuracy.
arXiv Detail & Related papers (2023-08-07T07:34:18Z) - Towards Compute-Optimal Transfer Learning [82.88829463290041]
We argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance.
Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.
arXiv Detail & Related papers (2023-04-25T21:49:09Z) - Deep Negative Correlation Classification [82.45045814842595]
Existing deep ensemble methods naively train many different models and then aggregate their predictions.
We propose deep negative correlation classification (DNCC)
DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
arXiv Detail & Related papers (2022-12-14T07:35:20Z) - A Mathematical Programming Approach to Optimal Classification Forests [1.0705399532413618]
We propose a novel mathematical optimization-based methodology in which a given number of trees are simultaneously constructed.
The classification rule is derived by assigning to each observation its most frequently predicted class among the trees in the forest.
We show that our proposed method has equal or superior performance compared with state-of-the-art tree-based classification methods.
arXiv Detail & Related papers (2022-11-18T20:33:08Z) - ProBoost: a Boosting Method for Probabilistic Classifiers [55.970609838687864]
ProBoost is a new boosting algorithm for probabilistic classifiers.
It uses the uncertainty of each training sample to determine the most challenging/uncertain ones.
It produces a sequence that progressively focuses on the samples found to have the highest uncertainty.
arXiv Detail & Related papers (2022-09-04T12:49:20Z) - Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios.
We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
arXiv Detail & Related papers (2022-04-13T12:43:12Z) - Improving the Accuracy-Memory Trade-Off of Random Forests Via
Leaf-Refinement [6.967385165474138]
Random Forests (RF) are among the state-of-the-art in many machine learning applications.
We show that the improvement effects of pruning diminish for ensembles of large trees but that pruning has an overall better accuracy-memory trade-off than RF.
We present a simple, yet surprisingly effective algorithm that refines the predictions in the leaf nodes in the forest via gradient descent.
arXiv Detail & Related papers (2021-10-19T16:06:43Z) - Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR
Prediction [19.71671771503269]
We develop a novel framework that adds regularizers of the sparse group lasso to a family of adaptives in deep learning.
We establish proven convergence guarantees in the theoretically convex settings.
Our methods can achieve extremely high sparsity with significantly better or highly competitive performance.
arXiv Detail & Related papers (2021-07-30T05:33:43Z) - Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
Variational autoencoders (VAE) are a powerful and widely-used class of generative models.
We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs.
Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
arXiv Detail & Related papers (2021-01-06T17:36:26Z) - Learning Diverse Representations for Fast Adaptation to Distribution
Shift [78.83747601814669]
We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task.
We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
arXiv Detail & Related papers (2020-06-12T12:23:50Z) - Sparsity in Optimal Randomized Classification Trees [3.441021278275805]
We propose a continuous optimization approach to build sparse optimal classification trees, based on oblique cuts.
Both types of sparsity, namely local and global, are modeled by means of regularizations with polyhedral norms.
Unlike greedy approaches, our ability to easily trade in some of our classification accuracy for a gain in global sparsity is shown.
arXiv Detail & Related papers (2020-02-21T09:09:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.