Explainable Models via Compression of Tree Ensembles

• URL: http://arxiv.org/abs/2206.07904v1
• Date: Thu, 16 Jun 2022 04:03:55 GMT
• Title: Explainable Models via Compression of Tree Ensembles
• Authors: Siwen Yan, Sriraam Natarajan, Saket Joshi, Roni Khardon and Prasad Tadepalli
• Abstract summary: We consider the problem of compressing a large set of learned trees into a single explainable model. CoTE -- Compression of Tree Ensembles -- produces a single small decision list as a compressed representation. An experimental evaluation demonstrates the effectiveness of CoTE in several benchmark relational data sets.
• Score: 23.790618990173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ensemble models (bagging and gradient-boosting) of relational decision trees have proved to be one of the most effective learning methods in the area of probabilistic logic models (PLMs). While effective, they lose one of the most important aspect of PLMs -- interpretability. In this paper we consider the problem of compressing a large set of learned trees into a single explainable model. To this effect, we propose CoTE -- Compression of Tree Ensembles -- that produces a single small decision list as a compressed representation. CoTE first converts the trees to decision lists and then performs the combination and compression with the aid of the original training set. An experimental evaluation demonstrates the effectiveness of CoTE in several benchmark relational data sets.

Related papers

• Decision Trees for Interpretable Clusters in Mixture Models and Deep Representations [5.65604054654671]
  We introduce the notion of an explainability-to-noise ratio for mixture models. We propose an algorithm that takes as input a mixture model and constructs a suitable tree in data-independent time. We prove upper and lower bounds on the error rate of the resulting decision tree.
  arXiv  Detail & Related papers  (2024-11-03T14:00:20Z)
• Inherently Interpretable Tree Ensemble Learning [7.868733904112288]
  We show that when shallow decision trees are used as base learners, the ensemble learning algorithms can become inherently interpretable. An interpretation algorithm is developed that converts the tree ensemble into the functional ANOVA representation with inherent interpretability. Experiments on simulations and real-world datasets show that our proposed methods offer a better trade-off between model interpretation and predictive performance.
  arXiv  Detail & Related papers  (2024-10-24T18:58:41Z)
• Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles [45.962492329047215]
  We introduce a method to prune a tree ensemble into a reduced version that is "functionally identical" to the original model. We formalize the problem of functionally identical pruning on ensembles, introduce an exact optimization model, and provide a fast yet highly effective method to prune large ensembles.
  arXiv  Detail & Related papers  (2024-08-28T23:15:46Z)
• Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers [68.76846801719095]
  We argue that the current high-level dichotomy into bias- and variance-reduction prevalent in statistics is insufficient to understand tree ensembles. We show that forests can improve upon trees by three distinct mechanisms that are usually implicitly entangled.
  arXiv  Detail & Related papers  (2024-02-02T15:36:43Z)
• A Robust Hypothesis Test for Tree Ensemble Pruning [2.4923006485141284]
  We develop and present a novel theoretically justified hypothesis test of split quality for gradient boosted tree ensembles. We show that using this method instead of the common penalty terms leads to a significant reduction in out of sample loss. We also present several innovative extensions to the method, opening the door for a wide variety of novel tree pruning algorithms.
  arXiv  Detail & Related papers  (2023-01-24T16:31:49Z)
• Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation [13.458414200958797]
  We conduct an empirical comparison of several previously-proposed methods for fair and robust learning alongside state-of-the-art tree-based methods. We show that tree-based methods have strong subgroup robustness, even when compared to robustness- and fairness-enhancing methods.
  arXiv  Detail & Related papers  (2022-11-23T04:49:18Z)
• TreeMix: Compositional Constituency-based Data Augmentation for Natural Language Understanding [56.794981024301094]
  We propose a compositional data augmentation approach for natural language understanding called TreeMix. Specifically, TreeMix leverages constituency parsing tree to decompose sentences into constituent sub-structures and the Mixup data augmentation technique to recombine them to generate new sentences. Compared with previous approaches, TreeMix introduces greater diversity to the samples generated and encourages models to learn compositionality of NLP data.
  arXiv  Detail & Related papers  (2022-05-12T15:25:12Z)
• Robustifying Algorithms of Learning Latent Trees with Vector Variables [92.18777020401484]
  We present the sample complexities of Recursive Grouping (RG) and Chow-Liu Recursive Grouping (CLRG) We robustify RG, CLRG, Neighbor Joining (NJ) and Spectral NJ (SNJ) by using the truncated inner product. We derive the first known instance-dependent impossibility result for structure learning of latent trees.
  arXiv  Detail & Related papers  (2021-06-02T01:37:52Z)
• Learning from Non-Binary Constituency Trees via Tensor Decomposition [12.069862650316262]
  We introduce a new approach to deal with non-binary constituency trees. We show how a powerful composition function based on the canonical tensor decomposition can exploit such a rich structure. We experimentally assess its performance on different NLP tasks.
  arXiv  Detail & Related papers  (2020-11-02T10:06:59Z)
• Convex Polytope Trees [57.56078843831244]
  convex polytope trees (CPT) are proposed to expand the family of decision trees by an interpretable generalization of their decision boundary. We develop a greedy method to efficiently construct CPT and scalable end-to-end training algorithms for the tree parameters when the tree structure is given.
  arXiv  Detail & Related papers  (2020-10-21T19:38:57Z)
• Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning [66.01622034708319]
  We propose a knowledge distillation based decision trees extension, dubbed rectified decision trees (ReDT) We extend the splitting criteria and the ending condition of the standard decision trees, which allows training with soft labels. We then train the ReDT based on the soft label distilled from a well-trained teacher model through a novel jackknife-based method.
  arXiv  Detail & Related papers  (2020-08-21T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.