Learning Sparse Decision Trees via Transformer Variational Auto-Encoders

• URL: http://arxiv.org/abs/2609.01430v1
• Date: Tue, 01 Sep 2026 15:40:33 GMT
• Title: Learning Sparse Decision Trees via Transformer Variational Auto-Encoders
• Abstract summary: We propose TREVIS, an approach for learning decision trees with respect to complex objectives.<n>By mapping decision trees onto latent representations, TREVIS replaces the discrete search space with a continuous one.<n>Results show that TREVIS discovers decision trees matching the predictive performance of existing near-optimal algorithms.
• Score: 2.2748974006378933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decision trees are among the most widely used models in machine learning, largely due to their transparent decision logic, making them well-suited for high-stakes decision-making contexts. However, most existing learning algorithms focus on predictive performance, overlooking the joint optimization of other desirable properties, such as structural sparsity. In this work we propose TREVIS, an approach for learning decision trees with respect to complex objectives, based on the exploration of the latent space of a Tree Transformer Variational Auto-Encoder (TTVAE). By mapping decision trees onto latent representations, TREVIS replaces the discrete search space with a continuous one, enabling gradient-based optimization via a differentiable surrogate model. We experiment with TREVIS for learning decision trees that jointly optimize predictive performance and sparsity. Results show that TREVIS discovers decision trees matching the predictive performance of existing near-optimal algorithms while improving their structural sparsity.

Related papers

• A novel gradient-based method for decision trees optimizing arbitrary differential loss functions [2.4861619769660637]
  This work introduces a novel method for constructing gradient-based decision trees that optimize arbitrary differentiable loss functions.<n>We demonstrate the method's applicability to classification, regression, and survival analysis tasks.<n>The implementation of the method is publicly available, providing a practical tool for researchers and practitioners.
  arXiv  Detail & Related papers  (2025-03-22T20:25:30Z)
• Decision Tree Induction Through LLMs via Semantically-Aware Evolution [53.0367886783772]
  We propose an evolutionary optimization method for decision tree induction based on genetic programming (GP)<n>Our key innovation is the integration of semantic priors and domain-specific knowledge about the search space into the algorithm.<n>This is operationalized through novel genetic operators that work with structured natural language prompts.
  arXiv  Detail & Related papers  (2025-03-18T12:52:03Z)
• WCDT: Systematic WCET Optimization for Decision Tree Implementations [4.95559363788634]
  Worst-case execution time (WCET) of machine-learning models is required to ensure safe operation.<n>We develop a systematic approach for WCET optimization of decision tree implementations.<n>We experimentally evaluate both the surrogate model and the WCET-optimization algorithm.
  arXiv  Detail & Related papers  (2025-01-29T06:01:39Z)
• Optimizing Interpretable Decision Tree Policies for Reinforcement Learning [10.68128849363198]
  Decision trees have gained increased attention in supervised learning for their inherent interpretability. This paper considers the problem of optimizing interpretable decision tree policies to replace neural networks in reinforcement learning settings.
  arXiv  Detail & Related papers  (2024-08-21T14:04:00Z)
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning [53.241569810013836]
  We propose a novel framework that utilizes large language models (LLMs) to identify effective feature generation rules. We use decision trees to convey this reasoning information, as they can be easily represented in natural language. OCTree consistently enhances the performance of various prediction models across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-06-12T08:31:34Z)
• Learning accurate and interpretable tree-based models [27.203303726977616]
  We develop approaches to design tree-based learning algorithms given repeated access to data from the same domain.<n>We propose novel parameterized classes of node splitting criteria in top-down algorithms, which interpolate between popularly used entropy and Gini impurity based criteria.<n>We extend our results to tuning popular tree-based ensembles, including random forests and gradient-boosted trees.
  arXiv  Detail & Related papers  (2024-05-24T20:10:10Z)
• An Interpretable Client Decision Tree Aggregation process for Federated Learning [7.8973037023478785]
  We propose an Interpretable Client Decision Tree aggregation process for Federated Learning scenarios. This model is based on aggregating multiple decision paths of the decision trees and can be used on different decision tree types, such as ID3 and CART. We carry out the experiments within four datasets, and the analysis shows that the tree built with the model improves the local models, and outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2024-04-03T06:53:56Z)
• ViTree: Single-path Neural Tree for Step-wise Interpretable Fine-grained Visual Categorization [56.37520969273242]
  We introduce ViTree, a novel approach for fine-grained visual categorization. By traversing the tree paths, ViTree effectively selects patches from transformer-processed features to highlight informative local regions. This patch and path selectivity enhances model interpretability of ViTree, enabling better insights into the model's inner workings.
  arXiv  Detail & Related papers  (2024-01-30T14:32:25Z)
• MurTree: Optimal Classification Trees via Dynamic Programming and Search [61.817059565926336]
  We present a novel algorithm for learning optimal classification trees based on dynamic programming and search. Our approach uses only a fraction of the time required by the state-of-the-art and can handle datasets with tens of thousands of instances.
  arXiv  Detail & Related papers  (2020-07-24T17:06:55Z)
• Generalized and Scalable Optimal Sparse Decision Trees [56.35541305670828]
  We present techniques that produce optimal decision trees over a variety of objectives. We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables.
  arXiv  Detail & Related papers  (2020-06-15T19:00:11Z)
• ENTMOOT: A Framework for Optimization over Ensemble Tree Models [57.98561336670884]
  ENTMOOT is a framework for integrating tree models into larger optimization problems. We show how ENTMOOT allows a simple integration of tree models into decision-making and black-box optimization.
  arXiv  Detail & Related papers  (2020-03-10T14:34:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.