Flexible Modeling and Multitask Learning using Differentiable Tree
Ensembles
- URL: http://arxiv.org/abs/2205.09717v1
- Date: Thu, 19 May 2022 17:30:49 GMT
- Title: Flexible Modeling and Multitask Learning using Differentiable Tree
Ensembles
- Authors: Shibal Ibrahim and Hussein Hazimeh and Rahul Mazumder
- Abstract summary: We propose a flexible framework for learning tree ensembles to support arbitrary loss functions, missing responses, and multi-task learning.
Our framework builds on differentiable tree ensembles, which can be trained using first-order methods.
We show that our framework can lead to 100x more compact and 23% more expressive tree ensembles than those by popular toolkits.
- Score: 6.037383467521294
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Decision tree ensembles are widely used and competitive learning models.
Despite their success, popular toolkits for learning tree ensembles have
limited modeling capabilities. For instance, these toolkits support a limited
number of loss functions and are restricted to single task learning. We propose
a flexible framework for learning tree ensembles, which goes beyond existing
toolkits to support arbitrary loss functions, missing responses, and multi-task
learning. Our framework builds on differentiable (a.k.a. soft) tree ensembles,
which can be trained using first-order methods. However, unlike classical
trees, differentiable trees are difficult to scale. We therefore propose a
novel tensor-based formulation of differentiable trees that allows for
efficient vectorization on GPUs. We perform experiments on a collection of 28
real open-source and proprietary datasets, which demonstrate that our framework
can lead to 100x more compact and 23% more expressive tree ensembles than those
by popular toolkits.
Related papers
- Terminating Differentiable Tree Experts [77.2443883991608]
We propose a neuro-symbolic Differentiable Tree Machine that learns tree operations using a combination of transformers and Representation Products.
We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts.
We additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically.
arXiv Detail & Related papers (2024-07-02T08:45:38Z) - Effective and Efficient Federated Tree Learning on Hybrid Data [80.31870543351918]
We propose HybridTree, a novel federated learning approach that enables federated tree learning on hybrid data.
We observe the existence of consistent split rules in trees and show that the knowledge of parties can be incorporated into the lower layers of a tree.
Our experiments demonstrate that HybridTree can achieve comparable accuracy to the centralized setting with low computational and communication overhead.
arXiv Detail & Related papers (2023-10-18T10:28:29Z) - Unboxing Tree Ensembles for interpretability: a hierarchical
visualization tool and a multivariate optimal re-built tree [0.34530027457862006]
We develop an interpretable representation of a tree-ensemble model that can provide valuable insights into its behavior.
The proposed model is effective in yielding a shallow interpretable tree approxing the tree-ensemble decision function.
arXiv Detail & Related papers (2023-02-15T10:43:31Z) - Highly Efficient Structural Learning of Sparse Staged Trees [2.3572498744567127]
We introduce the first scalable structural learning algorithm for staged trees, which searches over a space of models where only a small number of dependencies can be imposed.
A simulation study as well as a real-world application illustrate our routines and the practical use of such data-learned staged trees.
arXiv Detail & Related papers (2022-06-14T16:46:13Z) - Social Interpretable Tree for Pedestrian Trajectory Prediction [75.81745697967608]
We propose a tree-based method, termed as Social Interpretable Tree (SIT), to address this multi-modal prediction task.
A path in the tree from the root to leaf represents an individual possible future trajectory.
Despite the hand-crafted tree, the experimental results on ETH-UCY and Stanford Drone datasets demonstrate that our method is capable of matching or exceeding the performance of state-of-the-art methods.
arXiv Detail & Related papers (2022-05-26T12:18:44Z) - Incorporating Constituent Syntax for Coreference Resolution [50.71868417008133]
We propose a graph-based method to incorporate constituent syntactic structures.
We also explore to utilise higher-order neighbourhood information to encode rich structures in constituent trees.
Experiments on the English and Chinese portions of OntoNotes 5.0 benchmark show that our proposed model either beats a strong baseline or achieves new state-of-the-art performance.
arXiv Detail & Related papers (2022-02-22T07:40:42Z) - Shrub Ensembles for Online Classification [7.057937612386993]
Decision Tree (DT) ensembles provide excellent performance while adapting to changes in the data, but they are not resource efficient.
We propose a novel memory-efficient online classification ensemble called shrub ensembles for resource-constraint systems.
Our algorithm trains small to medium-sized decision trees on small windows and uses gradient descent to learn the ensemble weights of these shrubs'
arXiv Detail & Related papers (2021-12-07T14:22:43Z) - Dive into Decision Trees and Forests: A Theoretical Demonstration [0.0]
Decision trees use the strategy of "divide-and-conquer" to divide a complex problem on the dependency between input features and labels into smaller ones.
Recent advances have greatly improved their performance in computational advertising, recommender system, information retrieval, etc.
arXiv Detail & Related papers (2021-01-20T16:47:59Z) - Growing Deep Forests Efficiently with Soft Routing and Learned
Connectivity [79.83903179393164]
This paper further extends the deep forest idea in several important aspects.
We employ a probabilistic tree whose nodes make probabilistic routing decisions, a.k.a., soft routing, rather than hard binary decisions.
Experiments on the MNIST dataset demonstrate that our empowered deep forests can achieve better or comparable performance than [1],[3].
arXiv Detail & Related papers (2020-12-29T18:05:05Z) - MurTree: Optimal Classification Trees via Dynamic Programming and Search [61.817059565926336]
We present a novel algorithm for learning optimal classification trees based on dynamic programming and search.
Our approach uses only a fraction of the time required by the state-of-the-art and can handle datasets with tens of thousands of instances.
arXiv Detail & Related papers (2020-07-24T17:06:55Z) - Born-Again Tree Ensembles [9.307453801175177]
Tree ensembles offer a good prediction quality in various domains, but the concurrent use of multiple trees reduces the interpretability of the ensemble.
We study the process of constructing a single decision tree of minimum size that reproduces the exact same behavior as a given tree ensemble in its entire feature space.
This algorithm generates optimal born-again trees for many datasets of practical interest.
arXiv Detail & Related papers (2020-03-24T22:17:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.