Output-Constrained Decision Trees

• URL: http://arxiv.org/abs/2405.15314v3
• Date: Mon, 09 Jun 2025 08:25:17 GMT
• Title: Output-Constrained Decision Trees
• Authors: Hüseyin Tunç, Doğanay Özese, Ş. İlker Birbil, Donato Maragno, Marco Caserta, Mustafa Baydoğan,
• Abstract summary: This paper introduces new methods for training Output-Constrained Regression Trees (OCRT)<n>We propose three approaches: M-OCRT, which uses split-based mixed integer programming to enforce constraints; E-OCRT, which employs an exhaustive search for optimal splits and solves constrained prediction problems at each decision node; and EP-OCRT, which applies post-hoc constrained optimization to tree predictions.<n>Our results demonstrate that imposing constraints on decision tree training results in accurate and feasible predictions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Incorporating domain-specific constraints into machine learning models is essential for generating predictions that are both accurate and feasible in real-world applications. This paper introduces new methods for training Output-Constrained Regression Trees (OCRT), addressing the limitations of traditional decision trees in constrained multi-target regression tasks. We propose three approaches: M-OCRT, which uses split-based mixed integer programming to enforce constraints; E-OCRT, which employs an exhaustive search for optimal splits and solves constrained prediction problems at each decision node; and EP-OCRT, which applies post-hoc constrained optimization to tree predictions. To illustrate their potential uses in ensemble learning, we also introduce a random forest framework working under convex feasible sets. We validate the proposed methods through a computational study both on synthetic and industry-driven hierarchical time series datasets. Our results demonstrate that imposing constraints on decision tree training results in accurate and feasible predictions.

Related papers

• Extracting Interpretable Models from Tree Ensembles: Computational and Statistical Perspectives [14.726751780239907]
  We propose an estimator to extract compact sets of decision rules from tree ensembles.<n>A key novelty of our estimator is the flexibility to jointly control the number of rules extracted and the interaction depth of each rule.<n>We demonstrate that our estimator outperforms existing algorithms for rule extraction.
  arXiv  Detail & Related papers  (2025-06-25T04:06:37Z)
• A novel gradient-based method for decision trees optimizing arbitrary differential loss functions [2.4861619769660637]
  This work introduces a novel method for constructing gradient-based decision trees that optimize arbitrary differentiable loss functions.<n>We demonstrate the method's applicability to classification, regression, and survival analysis tasks.<n>The implementation of the method is publicly available, providing a practical tool for researchers and practitioners.
  arXiv  Detail & Related papers  (2025-03-22T20:25:30Z)
• Learning Decision Trees as Amortized Structure Inference [59.65621207449269]
  We propose a hybrid amortized structure inference approach to learn predictive decision tree ensembles given data.<n>We show that our approach, DT-GFN, outperforms state-of-the-art decision tree and deep learning methods on standard classification benchmarks.
  arXiv  Detail & Related papers  (2025-03-10T07:05:07Z)
• Multi-Objective Causal Bayesian Optimization [2.5311562666866494]
  We propose Multi-Objective Causal Bayesian Optimization (MO-CBO) to identify optimal interventions within a known multi-target causal graph. We show that MO-CBO can be decomposed into several traditional multi-objective optimization tasks. The proposed method will be validated on both synthetic and real-world causal graphs.
  arXiv  Detail & Related papers  (2025-02-20T17:26:16Z)
• A Unified Approach to Extract Interpretable Rules from Tree Ensembles via Integer Programming [2.1408617023874443]
  Tree ensembles are very popular machine learning models, known for their effectiveness in supervised classification and regression tasks.<n>Our work aims to extract an optimized list of rules from a trained tree ensemble, providing the user with a condensed, interpretable model that retains most of the predictive power of the full model.<n>Our extensive computational experiments offer statistically significant evidence that our method is competitive with other rule extraction methods in terms of predictive performance and fidelity towards the tree ensemble.
  arXiv  Detail & Related papers  (2024-06-30T22:33:47Z)
• Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment [103.12563033438715]
  Alignment in artificial intelligence pursues consistency between model responses and human preferences as well as values. Existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives.
  arXiv  Detail & Related papers  (2024-02-29T12:12:30Z)
• Deep Neural Network Benchmarks for Selective Classification [27.098996474946446]
  Multiple selective classification frameworks exist, most of which rely on deep neural network architectures. We evaluate these approaches using several criteria, including selective error rate, empirical coverage, distribution of rejected instance's classes, and performance on out-of-distribution instances.
  arXiv  Detail & Related papers  (2024-01-23T12:15:47Z)
• Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints [76.84999501420938]
  We introduce a conceptual and computational framework for assessing how the choice of target affects individuals' outcomes. We show that the level of multiplicity that stems from target variable choice can be greater than that stemming from nearly-optimal models of a single target.
  arXiv  Detail & Related papers  (2023-06-23T18:57:14Z)
• Best-Effort Adaptation [62.00856290846247]
  We present a new theoretical analysis of sample reweighting methods, including bounds holding uniformly over the weights. We show how these bounds can guide the design of learning algorithms that we discuss in detail. We report the results of a series of experiments demonstrating the effectiveness of our best-effort adaptation and domain adaptation algorithms.
  arXiv  Detail & Related papers  (2023-05-10T00:09:07Z)
• Multi-Target Decision Making under Conditions of Severe Uncertainty [0.0]
  We show how incomplete preferential and probabilistic information can be exploited to compare decisions among different targets. We discuss some interesting properties of the proposed orders between decision options and show how they can be concretely computed by linear optimization. We conclude the paper by demonstrating our framework in the context of comparing algorithms under different performance measures.
  arXiv  Detail & Related papers  (2022-12-13T11:47:02Z)
• Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation [0.0]
  Multi-Objective Policy Learning combines optimal decision trees for policy learning with a multi-objective Bayesian optimisation approach. The method is applied to a real-world case-study of non-price rationing of anti-malarial medication in Kenya.
  arXiv  Detail & Related papers  (2022-12-13T01:39:14Z)
• Semantic Probabilistic Layers for Neuro-Symbolic Learning [83.25785999205932]
  We design a predictive layer for structured-output prediction (SOP) It can be plugged into any neural network guaranteeing its predictions are consistent with a set of predefined symbolic constraints. Our Semantic Probabilistic Layer (SPL) can model intricate correlations, and hard constraints, over a structured output space.
  arXiv  Detail & Related papers  (2022-06-01T12:02:38Z)
• Generative multitask learning mitigates target-causing confounding [61.21582323566118]
  We propose a simple and scalable approach to causal representation learning for multitask learning. The improvement comes from mitigating unobserved confounders that cause the targets, but not the input. Our results on the Attributes of People and Taskonomy datasets reflect the conceptual improvement in robustness to prior probability shift.
  arXiv  Detail & Related papers  (2022-02-08T20:42:14Z)
• Robust Optimal Classification Trees Against Adversarial Examples [5.254093731341154]
  We propose a collection of methods to train decision trees that are optimally robust against user-specified attack models. We show that the min-max optimization problem that arises in adversarial learning can be solved using a single minimization formulation. We also present a method that determines the upper bound on adversarial accuracy for any model using bipartite matching.
  arXiv  Detail & Related papers  (2021-09-08T18:10:49Z)
• Counterfactual Maximum Likelihood Estimation for Training Deep Networks [83.44219640437657]
  Deep learning models are prone to learning spurious correlations that should not be learned as predictive clues. We propose a causality-based training framework to reduce the spurious correlations caused by observable confounders. We conduct experiments on two real-world tasks: Natural Language Inference (NLI) and Image Captioning.
  arXiv  Detail & Related papers  (2021-06-07T17:47:16Z)
• Efficient semidefinite-programming-based inference for binary and multi-class MRFs [83.09715052229782]
  We propose an efficient method for computing the partition function or MAP estimate in a pairwise MRF. We extend semidefinite relaxations from the typical binary MRF to the full multi-class setting, and develop a compact semidefinite relaxation that can again be solved efficiently using the solver.
  arXiv  Detail & Related papers  (2020-12-04T15:36:29Z)
• Combining Task Predictors via Enhancing Joint Predictability [53.46348489300652]
  We present a new predictor combination algorithm that improves the target by i) measuring the relevance of references based on their capabilities in predicting the target, and ii) strengthening such estimated relevance. Our algorithm jointly assesses the relevance of all references by adopting a Bayesian framework. Based on experiments on seven real-world datasets from visual attribute ranking and multi-class classification scenarios, we demonstrate that our algorithm offers a significant performance gain and broadens the application range of existing predictor combination approaches.
  arXiv  Detail & Related papers  (2020-07-15T21:58:39Z)
• An Integer Linear Programming Framework for Mining Constraints from Data [81.60135973848125]
  We present a general framework for mining constraints from data. In particular, we consider the inference in structured output prediction as an integer linear programming (ILP) problem. We show that our approach can learn to solve 9x9 Sudoku puzzles and minimal spanning tree problems from examples without providing the underlying rules.
  arXiv  Detail & Related papers  (2020-06-18T20:09:53Z)
• Generalized and Scalable Optimal Sparse Decision Trees [56.35541305670828]
  We present techniques that produce optimal decision trees over a variety of objectives. We also introduce a scalable algorithm that produces provably optimal results in the presence of continuous variables.
  arXiv  Detail & Related papers  (2020-06-15T19:00:11Z)
• Multivariate Boosted Trees and Applications to Forecasting and Control [0.0]
  Gradient boosted trees are non-parametric regressors that exploit sequential model fitting and gradient descent to minimize a specific loss function. In this paper, we present a computationally efficient algorithm for fitting multivariate boosted trees.
  arXiv  Detail & Related papers  (2020-03-08T19:26:59Z)
• Polynomial-Time Exact MAP Inference on Discrete Models with Global Dependencies [83.05591911173332]
  junction tree algorithm is the most general solution for exact MAP inference with run-time guarantees. We propose a new graph transformation technique via node cloning which ensures a run-time for solving our target problem independently of the form of a corresponding clique tree.
  arXiv  Detail & Related papers  (2019-12-27T13:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.