SMART: A Flexible Approach to Regression using Spline-Based Multivariate Adaptive Regression Trees

• URL: http://arxiv.org/abs/2410.05597v1
• Date: Tue, 8 Oct 2024 01:18:08 GMT
• Title: SMART: A Flexible Approach to Regression using Spline-Based Multivariate Adaptive Regression Trees
• Authors: William Pattie, Arvind Krishna,
• Abstract summary: Decision trees are powerful for predictive modeling but often suffer from high variance when modeling continuous relationships. We introduce Spline-based Multivariate Adaptive Regression Trees (MARS), which uses a decision tree to identify subsets of data with distinct continuous relationships. MARS's native ability to handle higher-order terms allows the tree to focus solely on identifying discontinuities in the relationship.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decision trees are powerful for predictive modeling but often suffer from high variance when modeling continuous relationships. While algorithms like Multivariate Adaptive Regression Splines (MARS) excel at capturing such continuous relationships, they perform poorly when modeling discontinuities. To address the limitations of both approaches, we introduce Spline-based Multivariate Adaptive Regression Trees (SMART), which uses a decision tree to identify subsets of data with distinct continuous relationships and then leverages MARS to fit these relationships independently. Unlike other methods that rely on the tree structure to model interaction and higher-order terms, SMART leverages MARS's native ability to handle these terms, allowing the tree to focus solely on identifying discontinuities in the relationship. We test SMART on various datasets, demonstrating its improvement over state-of-the-art methods in such cases. Additionally, we provide an open-source implementation of our method to be used by practitioners.

Related papers

• Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL [57.202733701029594]
  Decision Mamba is a novel multi-grained state space model with a self-evolving policy learning strategy. To mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations.
  arXiv  Detail & Related papers  (2024-06-08T10:12:00Z)
• An efficient solution to Hidden Markov Models on trees with coupled branches [0.0]
  We extend the framework of Hidden Models (HMMs) on trees to address scenarios where the tree-like structure of the data includes coupled branches. We develop a programming algorithm that efficiently solves the likelihood, decoding, and parameter learning problems for tree-based HMMs with coupled branches.
  arXiv  Detail & Related papers  (2024-06-03T18:00:00Z)
• Linear Mode Connectivity in Differentiable Tree Ensembles [13.704584231053675]
  Linear Mode Connectivity (LMC) refers to the phenomenon that remains consistent for linearly interpolated models in the parameter space. We first achieve LMC for soft tree neurons, which are tree-based differentiable models extensively used in practice.
  arXiv  Detail & Related papers  (2024-05-23T14:11:26Z)
• Multi-Target XGBoostLSS Regression [91.3755431537592]
  We present an extension of XGBoostLSS that models multiple targets and their dependencies in a probabilistic regression setting. Our approach outperforms existing GBMs with respect to runtime and compares well in terms of accuracy.
  arXiv  Detail & Related papers  (2022-10-13T08:26:14Z)
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression [0.0]
  We introduce TreeFlow, the tree-based approach that combines the benefits of using tree ensembles with the capabilities of modeling flexible probability distributions. We evaluate the proposed method on challenging regression benchmarks with varying volume, feature characteristics, and target dimensionality.
  arXiv  Detail & Related papers  (2022-06-08T20:06:23Z)
• Scalable Bayesian Network Structure Learning with Splines [2.741266294612776]
  A Bayesian Network (BN) is a probabilistic graphical model consisting of a directed acyclic graph (DAG) We present a novel approach capable of learning the global DAG structure of a BN and modelling linear and non-linear local relationships between variables.
  arXiv  Detail & Related papers  (2021-10-27T17:54:53Z)
• Trees-Based Models for Correlated Data [8.629912408966147]
  We show the problems that arise when implementing standard trees-based regression models, which ignore the correlation structure. Our new approach explicitly takes the correlation structure into account in the splitting criterion. The superiority of our new approach over trees-based models that do not account for the correlation is supported by simulation experiments and real data analyses.
  arXiv  Detail & Related papers  (2021-02-16T12:30:48Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• Estimation of Switched Markov Polynomial NARX models [75.91002178647165]
  We identify a class of models for hybrid dynamical systems characterized by nonlinear autoregressive (NARX) components. The proposed approach is demonstrated on a SMNARX problem composed by three nonlinear sub-models with specific regressors.
  arXiv  Detail & Related papers  (2020-09-29T15:00:47Z)
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation [77.7420231319632]
  We adapt contextual generation of categorical sequences to a policy gradient estimator, which evaluates a set of correlated Monte Carlo (MC) rollouts for variance control. We also demonstrate the use of correlated MC rollouts for binary-tree softmax models, which reduce the high generation cost in large vocabulary scenarios.
  arXiv  Detail & Related papers  (2019-12-31T03:01:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.