Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning

• URL: http://arxiv.org/abs/2008.09413v1
• Date: Fri, 21 Aug 2020 10:45:25 GMT
• Title: Rectified Decision Trees: Exploring the Landscape of Interpretable and Effective Machine Learning
• Authors: Yiming Li, Jiawang Bai, Jiawei Li, Xue Yang, Yong Jiang, Shu-Tao Xia
• Abstract summary: We propose a knowledge distillation based decision trees extension, dubbed rectified decision trees (ReDT) We extend the splitting criteria and the ending condition of the standard decision trees, which allows training with soft labels. We then train the ReDT based on the soft label distilled from a well-trained teacher model through a novel jackknife-based method.
• Score: 66.01622034708319
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interpretability and effectiveness are two essential and indispensable requirements for adopting machine learning methods in reality. In this paper, we propose a knowledge distillation based decision trees extension, dubbed rectified decision trees (ReDT), to explore the possibility of fulfilling those requirements simultaneously. Specifically, we extend the splitting criteria and the ending condition of the standard decision trees, which allows training with soft labels while preserving the deterministic splitting paths. We then train the ReDT based on the soft label distilled from a well-trained teacher model through a novel jackknife-based method. Accordingly, ReDT preserves the excellent interpretable nature of the decision trees while having a relatively good performance. The effectiveness of adopting soft labels instead of hard ones is also analyzed empirically and theoretically. Surprisingly, experiments indicate that the introduction of soft labels also reduces the model size compared with the standard decision trees from the aspect of the total nodes and rules, which is an unexpected gift from the `dark knowledge' distilled from the teacher model.

Related papers

• FoLDTree: A ULDA-Based Decision Tree Framework for Efficient Oblique Splits and Feature Selection [6.087464679182875]
  LDATree and FoLDTree integrate Uncorrelated Linear Discriminant Analysis (ULDA) and Forward ULDA into a decision tree structure. LDATree and FoLDTree consistently outperform axis-orthogonal and other oblique decision tree methods.
  arXiv  Detail & Related papers  (2024-10-30T16:03:51Z)
• Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers [68.76846801719095]
  We argue that the current high-level dichotomy into bias- and variance-reduction prevalent in statistics is insufficient to understand tree ensembles. We show that forests can improve upon trees by three distinct mechanisms that are usually implicitly entangled.
  arXiv  Detail & Related papers  (2024-02-02T15:36:43Z)
• Optimal Decision Diagrams for Classification [68.72078059880018]
  We study the training of optimal decision diagrams from a mathematical programming perspective. We introduce a novel mixed-integer linear programming model for training. We show how this model can be easily extended for fairness, parsimony, and stability notions.
  arXiv  Detail & Related papers  (2022-05-28T18:31:23Z)
• EiFFFeL: Enforcing Fairness in Forests by Flipping Leaves [5.08078625937586]
  We propose a fairness enforcing approach called EiFFFeL:Enforcing Fairness in Forests by Flipping Leaves. Experimental results show that our approach achieves a user defined group fairness degree without losing a significant amount of accuracy.
  arXiv  Detail & Related papers  (2021-12-29T07:48:38Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)
• Optimal randomized classification trees [0.0]
  Classification and Regression Trees (CARTs) are off-the-shelf techniques in modern Statistics and Machine Learning. CARTs are built by means of a greedy procedure, sequentially deciding the splitting predictor variable(s) and the associated threshold. This greedy approach trains trees very fast, but, by its nature, their classification accuracy may not be competitive against other state-of-the-art procedures.
  arXiv  Detail & Related papers  (2021-10-19T11:41:12Z)
• Making CNNs Interpretable by Building Dynamic Sequential Decision Forests with Top-down Hierarchy Learning [62.82046926149371]
  We propose a generic model transfer scheme to make Convlutional Neural Networks (CNNs) interpretable. We achieve this by building a differentiable decision forest on top of CNNs. We name the transferred model deep Dynamic Sequential Decision Forest (dDSDF)
  arXiv  Detail & Related papers  (2021-06-05T07:41:18Z)
• Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent [18.7724096545556]
  Decision trees provide a rich family of highly non-linear but efficient models. But learning trees is a challenging problem due to their highly discrete and non-differentiable decision boundaries. We propose a reformulation of the tree learning problem that provides better conditioned gradients.
  arXiv  Detail & Related papers  (2021-02-15T14:10:07Z)
• Fair Training of Decision Tree Classifiers [6.381149074212897]
  We study the problem of formally verifying individual fairness of decision tree ensembles. In our approach, fairness verification and fairness-aware training both rely on a notion of stability of a classification model.
  arXiv  Detail & Related papers  (2021-01-04T12:04:22Z)
• MurTree: Optimal Classification Trees via Dynamic Programming and Search [61.817059565926336]
  We present a novel algorithm for learning optimal classification trees based on dynamic programming and search. Our approach uses only a fraction of the time required by the state-of-the-art and can handle datasets with tens of thousands of instances.
  arXiv  Detail & Related papers  (2020-07-24T17:06:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.