Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression

• URL: http://arxiv.org/abs/2404.04498v2
• Date: Sat, 15 Jun 2024 06:58:56 GMT
• Title: Bayesian Inference for Consistent Predictions in Overparameterized Nonlinear Regression
• Authors: Tomoya Wakayama,
• Abstract summary: This study explores the predictive properties of over parameterized nonlinear regression within the Bayesian framework. Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions. The proposed method was validated via numerical simulations and a real data application.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The remarkable generalization performance of large-scale models has been challenging the conventional wisdom of the statistical learning theory. Although recent theoretical studies have shed light on this behavior in linear models and nonlinear classifiers, a comprehensive understanding of overparameterization in nonlinear regression models is still lacking. This study explores the predictive properties of overparameterized nonlinear regression within the Bayesian framework, extending the methodology of the adaptive prior considering the intrinsic spectral structure of the data. Posterior contraction is established for generalized linear and single-neuron models with Lipschitz continuous activation functions, demonstrating the consistency in the predictions of the proposed approach. Moreover, the Bayesian framework enables uncertainty estimation of the predictions. The proposed method was validated via numerical simulations and a real data application, showing its ability to achieve accurate predictions and reliable uncertainty estimates. This work provides a theoretical understanding of the advantages of overparameterization and a principled Bayesian approach to large nonlinear models.

Related papers

• Progression: an extrapolation principle for regression [0.0]
  We propose a novel statistical extrapolation principle. It assumes a simple relationship between predictors and the response at the boundary of the training predictor samples. Our semi-parametric method, progression, leverages this extrapolation principle and offers guarantees on the approximation error beyond the training data range.
  arXiv  Detail & Related papers  (2024-10-30T17:29:51Z)
• Mixture of partially linear experts [0.0]
  We propose a partially linear structure that incorporates unspecified functions to capture nonlinear relationships. We establish the identifiability of the proposed model under mild conditions and introduce a practical estimation algorithm.
  arXiv  Detail & Related papers  (2024-05-05T12:10:37Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Bayesian Analysis for Over-parameterized Linear Model without Sparsity [8.1585306387285]
  This study introduces a Bayesian approach that employs a prior distribution dependent on the eigenvectors of data covariance matrices without inducing parameter sparsity. We also provide contraction rates of the derived posterior estimation and develop a truncated Gaussian approximation of the posterior distribution. These findings suggest that Bayesian methods capable of handling data spectra and estimating non-sparse high-dimensional parameters are feasible.
  arXiv  Detail & Related papers  (2023-05-25T06:07:47Z)
• Time varying regression with hidden linear dynamics [74.9914602730208]
  We revisit a model for time-varying linear regression that assumes the unknown parameters evolve according to a linear dynamical system. Counterintuitively, we show that when the underlying dynamics are stable the parameters of this model can be estimated from data by combining just two ordinary least squares estimates.
  arXiv  Detail & Related papers  (2021-12-29T23:37:06Z)
• Hessian Eigenspectra of More Realistic Nonlinear Models [73.31363313577941]
  We make a emphprecise characterization of the Hessian eigenspectra for a broad family of nonlinear models. Our analysis takes a step forward to identify the origin of many striking features observed in more complex machine learning models.
  arXiv  Detail & Related papers  (2021-03-02T06:59:52Z)
• Weight-of-evidence 2.0 with shrinkage and spline-binning [3.925373521409752]
  We propose a formalized, data-driven and powerful method to transform categorical predictors. We extend upon the weight-of-evidence approach and propose to estimate the proportions using shrinkage estimators. We present the results of a series of experiments in a fraud detection setting, which illustrate the effectiveness of the presented approach.
  arXiv  Detail & Related papers  (2021-01-05T13:13:16Z)
• Assumption-lean inference for generalised linear model parameters [0.0]
  We propose nonparametric definitions of main effect estimands and effect modification estimands. These reduce to standard main effect and effect modification parameters in generalised linear models when these models are correctly specified. We achieve an assumption-lean inference for these estimands.
  arXiv  Detail & Related papers  (2020-06-15T13:49:48Z)
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation [74.06904875527556]
  We propose a generic and efficient segmentation framework to construct ensemble segmentation models. In the proposed method, ensemble models can be efficiently generated by using the layer selection method. We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
  arXiv  Detail & Related papers  (2020-05-21T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.