An Algorithmic Framework for Constructing Multiple Decision Trees by
Evaluating Their Combination Performance Throughout the Construction Process
- URL: http://arxiv.org/abs/2402.06452v1
- Date: Fri, 9 Feb 2024 14:58:07 GMT
- Title: An Algorithmic Framework for Constructing Multiple Decision Trees by
Evaluating Their Combination Performance Throughout the Construction Process
- Authors: Keito Tajima, Naoki Ichijo, Yuta Nakahara, and Toshiyasu Matsushima
- Abstract summary: Predictions using a combination of decision trees are known to be effective in machine learning.
We propose a new algorithmic framework that constructs decision trees simultaneously and evaluates their combination performance.
- Score: 1.8749305679160366
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Predictions using a combination of decision trees are known to be effective
in machine learning. Typical ideas for constructing a combination of decision
trees for prediction are bagging and boosting. Bagging independently constructs
decision trees without evaluating their combination performance and averages
them afterward. Boosting constructs decision trees sequentially, only
evaluating a combination performance of a new decision tree and the fixed past
decision trees at each step. Therefore, neither method directly constructs nor
evaluates a combination of decision trees for the final prediction. When the
final prediction is based on a combination of decision trees, it is natural to
evaluate the appropriateness of the combination when constructing them. In this
study, we propose a new algorithmic framework that constructs decision trees
simultaneously and evaluates their combination performance throughout the
construction process. Our framework repeats two procedures. In the first
procedure, we construct new candidates of combinations of decision trees to
find a proper combination of decision trees. In the second procedure, we
evaluate each combination performance of decision trees under some criteria and
select a better combination. To confirm the performance of the proposed
framework, we perform experiments on synthetic and benchmark data.
Related papers
- Learning accurate and interpretable decision trees [27.203303726977616]
We develop approaches to design decision tree learning algorithms given repeated access to data from the same domain.
We study the sample complexity of tuning prior parameters in Bayesian decision tree learning, and extend our results to decision tree regression.
We also study the interpretability of the learned decision trees and introduce a data-driven approach for optimizing the explainability versus accuracy trade-off using decision trees.
arXiv Detail & Related papers (2024-05-24T20:10:10Z) - Divide, Conquer, Combine Bayesian Decision Tree Sampling [1.1879716317856945]
Decision trees are commonly used predictive models due to their flexibility and interpretability.
This paper is directed at quantifying the uncertainty of decision tree predictions by employing a Bayesian inference approach.
arXiv Detail & Related papers (2024-03-26T23:14:15Z) - Boosting-Based Sequential Meta-Tree Ensemble Construction for Improved
Decision Trees [1.8749305679160366]
A decision tree is one of the most popular approaches in machine learning fields.
A meta-tree is recently proposed to solve the problem of overfitting caused by overly deepened trees.
The meta-tree guarantees statistical optimality based on Bayes decision theory.
arXiv Detail & Related papers (2024-02-09T13:08:21Z) - Learning a Decision Tree Algorithm with Transformers [75.96920867382859]
We introduce MetaTree, a transformer-based model trained via meta-learning to directly produce strong decision trees.
We fit both greedy decision trees and globally optimized decision trees on a large number of datasets, and train MetaTree to produce only the trees that achieve strong generalization performance.
arXiv Detail & Related papers (2024-02-06T07:40:53Z) - TreeDQN: Learning to minimize Branch-and-Bound tree [78.52895577861327]
Branch-and-Bound is a convenient approach to solving optimization tasks in the form of Mixed Linear Programs.
The efficiency of the solver depends on the branchning used to select a variable for splitting.
We propose a reinforcement learning method that can efficiently learn the branching.
arXiv Detail & Related papers (2023-06-09T14:01:26Z) - Contextual Decision Trees [62.997667081978825]
We propose a multi-armed contextual bandit recommendation framework for feature-based selection of a single shallow tree of the learned ensemble.
The trained system, which works on top of the Random Forest, dynamically identifies a base predictor that is responsible for providing the final output.
arXiv Detail & Related papers (2022-07-13T17:05:08Z) - Optimal randomized classification trees [0.0]
Classification and Regression Trees (CARTs) are off-the-shelf techniques in modern Statistics and Machine Learning.
CARTs are built by means of a greedy procedure, sequentially deciding the splitting predictor variable(s) and the associated threshold.
This greedy approach trains trees very fast, but, by its nature, their classification accuracy may not be competitive against other state-of-the-art procedures.
arXiv Detail & Related papers (2021-10-19T11:41:12Z) - Stochastic Optimization Forests [60.523606291705214]
We show how to train forest decision policies by growing trees that choose splits to directly optimize the downstream decision quality, rather than splitting to improve prediction accuracy as in the standard random forest algorithm.
We show that our approximate splitting criteria can reduce running time hundredfold, while achieving performance close to forest algorithms that exactly re-optimize for every candidate split.
arXiv Detail & Related papers (2020-08-17T16:56:06Z) - MurTree: Optimal Classification Trees via Dynamic Programming and Search [61.817059565926336]
We present a novel algorithm for learning optimal classification trees based on dynamic programming and search.
Our approach uses only a fraction of the time required by the state-of-the-art and can handle datasets with tens of thousands of instances.
arXiv Detail & Related papers (2020-07-24T17:06:55Z) - Generalized and Scalable Optimal Sparse Decision Trees [56.35541305670828]
We present techniques that produce optimal decision trees over a variety of objectives.
We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables.
arXiv Detail & Related papers (2020-06-15T19:00:11Z) - Evolutionary algorithms for constructing an ensemble of decision trees [0.0]
We propose several methods for induction of decision trees and their ensembles based on evolutionary algorithms.
The main difference of our approach is using real-valued vector representation of decision tree.
We test the predictive performance of this methods using several public UCI data sets.
arXiv Detail & Related papers (2020-02-03T13:38:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.