MARS via LASSO

• URL: http://arxiv.org/abs/2111.11694v5
• Date: Sun, 13 Oct 2024 21:12:36 GMT
• Title: MARS via LASSO
• Authors: Dohyeong Ki, Billy Fang, Adityanand Guntuboyina,
• Abstract summary: We propose and study a natural variant of the MARS method. Our method is based on at least squares estimation over a convex class of functions. Our estimator can be computed via finite-dimensional convex optimization.
• Score: 1.5199437137239338
• License:
• Abstract: Multivariate adaptive regression splines (MARS) is a popular method for nonparametric regression introduced by Friedman in 1991. MARS fits simple nonlinear and non-additive functions to regression data. We propose and study a natural lasso variant of the MARS method. Our method is based on least squares estimation over a convex class of functions obtained by considering infinite-dimensional linear combinations of functions in the MARS basis and imposing a variation based complexity constraint. Our estimator can be computed via finite-dimensional convex optimization, although it is defined as a solution to an infinite-dimensional optimization problem. Under a few standard design assumptions, we prove that our estimator achieves a rate of convergence that depends only logarithmically on dimension and thus avoids the usual curse of dimensionality to some extent. We also show that our method is naturally connected to nonparametric estimation techniques based on smoothness constraints. We implement our method with a cross-validation scheme for the selection of the involved tuning parameter and compare it to the usual MARS method in various simulation and real data settings.

Related papers

• Total Uncertainty Quantification in Inverse PDE Solutions Obtained with Reduced-Order Deep Learning Surrogate Models [50.90868087591973]
  We propose an approximate Bayesian method for quantifying the total uncertainty in inverse PDE solutions obtained with machine learning surrogate models. We test the proposed framework by comparing it with the iterative ensemble smoother and deep ensembling methods for a non-linear diffusion equation.
  arXiv  Detail & Related papers  (2024-08-20T19:06:02Z)
• Dimension Reduction and MARS [4.525349089861123]
  The adaptive regression spline (MARS) is one of the popular estimation methods for nonparametric multivariate regressions. In this paper, we improve the performance of MARS by using linear combinations of the covariates which achieve sufficient dimension reduction. Numerical studies and empirical applications show its effectiveness and improvement over MARS and other commonly-used nonparametric methods in regression estimation and prediction.
  arXiv  Detail & Related papers  (2023-02-11T21:50:17Z)
• Riemannian Optimization for Variance Estimation in Linear Mixed Models [0.0]
  We take a completely novel view on parameter estimation in linear mixed models by exploiting the intrinsic geometry of the parameter space. Our approach yields a higher quality of the variance parameter estimates compared to existing approaches.
  arXiv  Detail & Related papers  (2022-12-18T13:08:45Z)
• Manifold Gaussian Variational Bayes on the Precision Matrix [70.44024861252554]
  We propose an optimization algorithm for Variational Inference (VI) in complex models. We develop an efficient algorithm for Gaussian Variational Inference whose updates satisfy the positive definite constraint on the variational covariance matrix. Due to its black-box nature, MGVBP stands as a ready-to-use solution for VI in complex models.
  arXiv  Detail & Related papers  (2022-10-26T10:12:31Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Sparse high-dimensional linear regression with a partitioned empirical Bayes ECM algorithm [62.997667081978825]
  We propose a computationally efficient and powerful Bayesian approach for sparse high-dimensional linear regression. Minimal prior assumptions on the parameters are used through the use of plug-in empirical Bayes estimates. The proposed approach is implemented in the R package probe.
  arXiv  Detail & Related papers  (2022-09-16T19:15:50Z)
• Robust Regression via Model Based Methods [13.300549123177705]
  We propose an algorithm inspired by so-called model-based optimization (MBO) [35, 36], which replaces a non-objective with a convex model function. We apply this to robust regression, proposing SADM, a function of the Online Alternating Direction Method of Multipliers (OOADM) [50] to solve the inner optimization in MBO. Finally, we demonstrate experimentally (a) the robustness of l_p norms to outliers and (b) the efficiency of our proposed model-based algorithms in comparison with methods on autoencoders and multi-target regression.
  arXiv  Detail & Related papers  (2021-06-20T21:45:35Z)
• Parallel Stochastic Mirror Descent for MDPs [72.75921150912556]
  We consider the problem of learning the optimal policy for infinite-horizon Markov decision processes (MDPs) Some variant of Mirror Descent is proposed for convex programming problems with Lipschitz-continuous functionals. We analyze this algorithm in a general case and obtain an estimate of the convergence rate that does not accumulate errors during the operation of the method.
  arXiv  Detail & Related papers  (2021-02-27T19:28:39Z)
• Reduction of the Number of Variables in Parametric Constrained Least-Squares Problems [0.20305676256390928]
  This paper proposes techniques for reducing the number of involved optimization variables. We show the good performance of the proposed techniques in numerical tests and in a linearized MPC problem of a nonlinear benchmark process.
  arXiv  Detail & Related papers  (2020-12-18T18:26:40Z)
• Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization [56.05635751529922]
  We propose a new randomized algorithm for solving L2-regularized least-squares problems based on sketching. We consider two of the most popular random embeddings, namely, Gaussian embeddings and the Subsampled Randomized Hadamard Transform (SRHT)
  arXiv  Detail & Related papers  (2020-06-10T15:00:09Z)
• FREDE: Linear-Space Anytime Graph Embeddings [12.53022591889574]
  Low-dimensional representations, or embeddings, of a graph's nodes facilitate data mining tasks. FREquent Directions Embedding is a sketching-based method that iteratively improves on quality while processing rows of the similarity matrix individually. Our evaluation on variably sized networks shows that FREDE performs as well as SVD and competitively against current state-of-the-art methods in diverse data mining tasks.
  arXiv  Detail & Related papers  (2020-06-08T16:51:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.