Explainable Models via Compression of Tree Ensembles
- URL: http://arxiv.org/abs/2206.07904v1
- Date: Thu, 16 Jun 2022 04:03:55 GMT
- Title: Explainable Models via Compression of Tree Ensembles
- Authors: Siwen Yan, Sriraam Natarajan, Saket Joshi, Roni Khardon and Prasad
Tadepalli
- Abstract summary: We consider the problem of compressing a large set of learned trees into a single explainable model.
CoTE -- Compression of Tree Ensembles -- produces a single small decision list as a compressed representation.
An experimental evaluation demonstrates the effectiveness of CoTE in several benchmark relational data sets.
- Score: 23.790618990173
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Ensemble models (bagging and gradient-boosting) of relational decision trees
have proved to be one of the most effective learning methods in the area of
probabilistic logic models (PLMs). While effective, they lose one of the most
important aspect of PLMs -- interpretability. In this paper we consider the
problem of compressing a large set of learned trees into a single explainable
model. To this effect, we propose CoTE -- Compression of Tree Ensembles -- that
produces a single small decision list as a compressed representation. CoTE
first converts the trees to decision lists and then performs the combination
and compression with the aid of the original training set. An experimental
evaluation demonstrates the effectiveness of CoTE in several benchmark
relational data sets.
Related papers
- Why do Random Forests Work? Understanding Tree Ensembles as
Self-Regularizing Adaptive Smoothers [68.76846801719095]
We argue that the current high-level dichotomy into bias- and variance-reduction prevalent in statistics is insufficient to understand tree ensembles.
We show that forests can improve upon trees by three distinct mechanisms that are usually implicitly entangled.
arXiv Detail & Related papers (2024-02-02T15:36:43Z) - A Robust Hypothesis Test for Tree Ensemble Pruning [2.4923006485141284]
We develop and present a novel theoretically justified hypothesis test of split quality for gradient boosted tree ensembles.
We show that using this method instead of the common penalty terms leads to a significant reduction in out of sample loss.
We also present several innovative extensions to the method, opening the door for a wide variety of novel tree pruning algorithms.
arXiv Detail & Related papers (2023-01-24T16:31:49Z) - Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation [13.458414200958797]
We conduct an empirical comparison of several previously-proposed methods for fair and robust learning alongside state-of-the-art tree-based methods.
We show that tree-based methods have strong subgroup robustness, even when compared to robustness- and fairness-enhancing methods.
arXiv Detail & Related papers (2022-11-23T04:49:18Z) - TreeMix: Compositional Constituency-based Data Augmentation for Natural
Language Understanding [56.794981024301094]
We propose a compositional data augmentation approach for natural language understanding called TreeMix.
Specifically, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmentation technique to recombine them to generate new sentences.
Compared with previous approaches, TreeMix introduces greater diversity to the samples generated and encourages models to learn compositionality of NLP data.
arXiv Detail & Related papers (2022-05-12T15:25:12Z) - A cautionary tale on fitting decision trees to data from additive
models: generalization lower bounds [9.546094657606178]
We study the generalization performance of decision trees with respect to different generative regression models.
This allows us to elicit their inductive bias, that is, the assumptions the algorithms make (or do not make) to generalize to new data.
We prove a sharp squared error generalization lower bound for a large class of decision tree algorithms fitted to sparse additive models.
arXiv Detail & Related papers (2021-10-18T21:22:40Z) - Robust Optimal Classification Trees Against Adversarial Examples [5.254093731341154]
We propose a collection of methods to train decision trees that are optimally robust against user-specified attack models.
We show that the min-max optimization problem that arises in adversarial learning can be solved using a single minimization formulation.
We also present a method that determines the upper bound on adversarial accuracy for any model using bipartite matching.
arXiv Detail & Related papers (2021-09-08T18:10:49Z) - Robustifying Algorithms of Learning Latent Trees with Vector Variables [92.18777020401484]
We present the sample complexities of Recursive Grouping (RG) and Chow-Liu Recursive Grouping (CLRG)
We robustify RG, CLRG, Neighbor Joining (NJ) and Spectral NJ (SNJ) by using the truncated inner product.
We derive the first known instance-dependent impossibility result for structure learning of latent trees.
arXiv Detail & Related papers (2021-06-02T01:37:52Z) - Learning from Non-Binary Constituency Trees via Tensor Decomposition [12.069862650316262]
We introduce a new approach to deal with non-binary constituency trees.
We show how a powerful composition function based on the canonical tensor decomposition can exploit such a rich structure.
We experimentally assess its performance on different NLP tasks.
arXiv Detail & Related papers (2020-11-02T10:06:59Z) - An Efficient Adversarial Attack for Tree Ensembles [91.05779257472675]
adversarial attacks on tree based ensembles such as gradient boosting decision trees (DTs) and random forests (RFs)
We show that our method can be thousands of times faster than the previous mixed-integer linear programming (MILP) based approach.
Our code is available at https://chong-z/tree-ensemble-attack.
arXiv Detail & Related papers (2020-10-22T10:59:49Z) - Convex Polytope Trees [57.56078843831244]
convex polytope trees (CPT) are proposed to expand the family of decision trees by an interpretable generalization of their decision boundary.
We develop a greedy method to efficiently construct CPT and scalable end-to-end training algorithms for the tree parameters when the tree structure is given.
arXiv Detail & Related papers (2020-10-21T19:38:57Z) - Rectified Decision Trees: Exploring the Landscape of Interpretable and
Effective Machine Learning [66.01622034708319]
We propose a knowledge distillation based decision trees extension, dubbed rectified decision trees (ReDT)
We extend the splitting criteria and the ending condition of the standard decision trees, which allows training with soft labels.
We then train the ReDT based on the soft label distilled from a well-trained teacher model through a novel jackknife-based method.
arXiv Detail & Related papers (2020-08-21T10:45:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.