Using Model-Based Trees with Boosting to Fit Low-Order Functional ANOVA Models

• URL: http://arxiv.org/abs/2207.06950v5
• Date: Fri, 15 Dec 2023 19:20:55 GMT
• Title: Using Model-Based Trees with Boosting to Fit Low-Order Functional ANOVA Models
• Authors: Linwei Hu, Jie Chen, Vijayan N. Nair
• Abstract summary: Low-order functional ANOVA models have been rediscovered in the machine learning (ML) community under the guise of inherently interpretable machine learning. We propose a new algorithm, called GAMI-Tree, that is similar to EBM, but has a number of features that lead to better performance. We use simulated and real datasets to compare the performance and interpretability of GAMI-Tree with EBM and GAMI-Net.
• Score: 5.131758478675364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-order functional ANOVA (fANOVA) models have been rediscovered in the machine learning (ML) community under the guise of inherently interpretable machine learning. Explainable Boosting Machines or EBM (Lou et al. 2013) and GAMI-Net (Yang et al. 2021) are two recently proposed ML algorithms for fitting functional main effects and second-order interactions. We propose a new algorithm, called GAMI-Tree, that is similar to EBM, but has a number of features that lead to better performance. It uses model-based trees as base learners and incorporates a new interaction filtering method that is better at capturing the underlying interactions. In addition, our iterative training method converges to a model with better predictive performance, and the embedded purification ensures that interactions are hierarchically orthogonal to main effects. The algorithm does not need extensive tuning, and our implementation is fast and efficient. We use simulated and real datasets to compare the performance and interpretability of GAMI-Tree with EBM and GAMI-Net.

Related papers

• Monotone Tree-Based GAMI Models by Adapting XGBoost [4.566028525473582]
  This paper considers models of the form $f(x)=sum_j,kf_j,k(x_j, x_k)$ and develops monotone tree-based GAMI models, called monotone GAMI-Tree. It is straightforward to fit a monotone model to $f(x)$ using the options in XGBoost. However, the fitted model is still a black box. We take a different approach: i) use a filtering technique to determine the important interactions, ii) fit a monotone XGBoost algorithm with the selected interactions, and
  arXiv  Detail & Related papers  (2023-09-05T17:54:37Z)
• Interpretable Machine Learning based on Functional ANOVA Framework: Algorithms and Comparisons [9.10422407200807]
  In the early days of machine learning (ML), the emphasis was on developing complex algorithms to achieve best predictive performance. Recently, researchers are compromising on small increases in predictive performance to develop algorithms that are inherently interpretable. The paper proposes a new algorithm, called GAMI-Lin-T, that also uses trees like EBM, but it does linear fits instead of piecewise constants within the partitions.
  arXiv  Detail & Related papers  (2023-05-25T02:40:52Z)
• Unboxing Tree Ensembles for interpretability: a hierarchical visualization tool and a multivariate optimal re-built tree [0.34530027457862006]
  We develop an interpretable representation of a tree-ensemble model that can provide valuable insights into its behavior. The proposed model is effective in yielding a shallow interpretable tree approxing the tree-ensemble decision function.
  arXiv  Detail & Related papers  (2023-02-15T10:43:31Z)
• Sparse MoEs meet Efficient Ensembles [49.313497379189315]
  We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs) We present Efficient Ensemble of Experts (E$3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble.
  arXiv  Detail & Related papers  (2021-10-07T11:58:35Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Adapting by Pruning: A Case Study on BERT [9.963251767416967]
  We propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task. We formulate adapting-by-pruning as an optimisation problem with a differentiable loss and propose an efficient algorithm to prune the model. Results suggest that our method can prune up to 50% weights in BERT while yielding similar performance compared to the fine-tuned full model.
  arXiv  Detail & Related papers  (2021-05-07T15:51:08Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Surrogate Locally-Interpretable Models with Supervised Machine Learning Algorithms [8.949704905866888]
  Supervised Machine Learning algorithms have become popular in recent years due to their superior predictive performance over traditional statistical methods. The main focus is on interpretability, the resulting surrogate model also has reasonably good predictive performance.
  arXiv  Detail & Related papers  (2020-07-28T23:46:16Z)
• Interpretable Learning-to-Rank with Generalized Additive Models [78.42800966500374]
  Interpretability of learning-to-rank models is a crucial yet relatively under-examined research area. Recent progress on interpretable ranking models largely focuses on generating post-hoc explanations for existing black-box ranking models. We lay the groundwork for intrinsically interpretable learning-to-rank by introducing generalized additive models (GAMs) into ranking tasks.
  arXiv  Detail & Related papers  (2020-05-06T01:51:30Z)
• MPLP++: Fast, Parallel Dual Block-Coordinate Ascent for Dense Graphical Models [96.1052289276254]
  This work introduces a new MAP-solver, based on the popular Dual Block-Coordinate Ascent principle. Surprisingly, by making a small change to the low-performing solver, we derive the new solver MPLP++ that significantly outperforms all existing solvers by a large margin.
  arXiv  Detail & Related papers  (2020-04-16T16:20:53Z)
• Revisiting Graph based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach [55.44107800525776]
  Graph Convolutional Networks (GCNs) are state-of-the-art graph based representation learning models. In this paper, we revisit GCN based Collaborative Filtering (CF) based Recommender Systems (RS) We show that removing non-linearities would enhance recommendation performance, consistent with the theories in simple graph convolutional networks. We propose a residual network structure that is specifically designed for CF with user-item interaction modeling.
  arXiv  Detail & Related papers  (2020-01-28T04:41:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.