Nested Model Averaging on Solution Path for High-dimensional Linear Regression

• URL: http://arxiv.org/abs/2005.08057v1
• Date: Sat, 16 May 2020 18:09:57 GMT
• Title: Nested Model Averaging on Solution Path for High-dimensional Linear Regression
• Authors: Yang Feng and Qingfeng Liu
• Abstract summary: We study the nested model averaging method on the solution path for a high-dimensional linear regression problem. We propose to combine model averaging with regularized estimators (e.g., lasso and SLOPE) on the solution path for high-dimensional linear regression. A real data analysis on predicting the per capita violent crime in the United States shows an outstanding performance of the nested model averaging with lasso.
• Score: 12.173071615025504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the nested model averaging method on the solution path for a high-dimensional linear regression problem. In particular, we propose to combine model averaging with regularized estimators (e.g., lasso and SLOPE) on the solution path for high-dimensional linear regression. In simulation studies, we first conduct a systematic investigation on the impact of predictor ordering on the behavior of nested model averaging, then show that nested model averaging with lasso and SLOPE compares favorably with other competing methods, including the infeasible lasso and SLOPE with the tuning parameter optimally selected. A real data analysis on predicting the per capita violent crime in the United States shows an outstanding performance of the nested model averaging with lasso.

Related papers

• Conformal prediction of circular data [1.6385815610837167]
  Split conformal prediction techniques are applied to regression problems with circular responses. We analyze a general projection procedure that converts any linear response regression model into one suitable for circular responses.
  arXiv  Detail & Related papers  (2024-10-31T17:05:52Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• Adaptive Optimization for Prediction with Missing Data [6.800113478497425]
  We show that some adaptive linear regression models are equivalent to learning an imputation rule and a downstream linear regression model simultaneously. In settings where data is strongly not missing at random, our methods achieve a 2-10% improvement in out-of-sample accuracy.
  arXiv  Detail & Related papers  (2024-02-02T16:35:51Z)
• Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift [12.770658031721435]
  We propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution. We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-12-29T04:15:58Z)
• Variational Laplace Autoencoders [53.08170674326728]
  Variational autoencoders employ an amortized inference model to approximate the posterior of latent variables. We present a novel approach that addresses the limited posterior expressiveness of fully-factorized Gaussian assumption. We also present a general framework named Variational Laplace Autoencoders (VLAEs) for training deep generative models.
  arXiv  Detail & Related papers  (2022-11-30T18:59:27Z)
• An adaptive shortest-solution guided decimation approach to sparse high-dimensional linear regression [2.3759847811293766]
  ASSD is adapted from the shortest solution-guided algorithm and is referred to as ASSD. ASSD is especially suitable for linear regression problems with highly correlated measurement matrices encountered in real-world applications.
  arXiv  Detail & Related papers  (2022-11-28T04:29:57Z)
• Time varying regression with hidden linear dynamics [74.9914602730208]
  We revisit a model for time-varying linear regression that assumes the unknown parameters evolve according to a linear dynamical system. Counterintuitively, we show that when the underlying dynamics are stable the parameters of this model can be estimated from data by combining just two ordinary least squares estimates.
  arXiv  Detail & Related papers  (2021-12-29T23:37:06Z)
• An Extended Multi-Model Regression Approach for Compressive Strength Prediction and Optimization of a Concrete Mixture [0.0]
  A model based evaluation of concrete compressive strength is of high value, both for the purpose of strength prediction and the mixture optimization. We take a further step towards improving the accuracy of the prediction model via the weighted combination of multiple regression methods. A proposed (GA)-based mixture optimization is proposed, building on the obtained multi-regression model.
  arXiv  Detail & Related papers  (2021-06-13T16:10:32Z)
• Estimation of Switched Markov Polynomial NARX models [75.91002178647165]
  We identify a class of models for hybrid dynamical systems characterized by nonlinear autoregressive (NARX) components. The proposed approach is demonstrated on a SMNARX problem composed by three nonlinear sub-models with specific regressors.
  arXiv  Detail & Related papers  (2020-09-29T15:00:47Z)
• Model Fusion with Kullback--Leibler Divergence [58.20269014662046]
  We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors.
  arXiv  Detail & Related papers  (2020-07-13T03:27:45Z)
• Control as Hybrid Inference [62.997667081978825]
  We present an implementation of CHI which naturally mediates the balance between iterative and amortised inference. We verify the scalability of our algorithm on a continuous control benchmark, demonstrating that it outperforms strong model-free and model-based baselines.
  arXiv  Detail & Related papers  (2020-07-11T19:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.