Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression

• URL: http://arxiv.org/abs/2306.03783v2
• Date: Thu, 26 Oct 2023 18:53:58 GMT
• Title: Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression
• Authors: Youngsoo Baek, Samuel I. Berchuck, Sayan Mukherjee
• Abstract summary: We focus on the variance of the posterior predictive distribution (Bayesian model average) and compare itss to that of the risk of the MAP estimator. They also agree with each other when the number of samples grow faster than any constant multiple of model dimensions.
• Score: 1.170951597793276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper we compare and contrast the behavior of the posterior predictive distribution to the risk of the maximum a posteriori estimator for the random features regression model in the overparameterized regime. We will focus on the variance of the posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grow faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, which is of independent theoretical interest.

Related papers

• Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis [56.442307356162864]
  We study the theoretical aspects of score-based discrete diffusion models under the Continuous Time Markov Chain (CTMC) framework. We introduce a discrete-time sampling algorithm in the general state space $[S]d$ that utilizes score estimators at predefined time points. Our convergence analysis employs a Girsanov-based method and establishes key properties of the discrete score function.
  arXiv  Detail & Related papers  (2024-10-03T09:07:13Z)
• Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent [6.3151583550712065]
  We study the dynamics of a continuous-time model of the Gradient Descent (SGD) We analyze degenerate Differential Equations (squareSDEs) that model SGD either in the case of the training loss (finite samples) or the population one (online setting)
  arXiv  Detail & Related papers  (2024-07-02T14:52:21Z)
• von Mises Quasi-Processes for Bayesian Circular Regression [57.88921637944379]
  We explore a family of expressive and interpretable distributions over circle-valued random functions. The resulting probability model has connections with continuous spin models in statistical physics. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Variational excess risk bound for general state space models [0.0]
  We consider variational autoencoders (VAE) for general state space models. We consider a backward factorization of the variational distributions to analyze the excess risk associated with VAE.
  arXiv  Detail & Related papers  (2023-12-15T08:41:07Z)
• Conformal inference for regression on Riemannian Manifolds [49.7719149179179]
  We investigate prediction sets for regression scenarios when the response variable, denoted by $Y$, resides in a manifold, and the covariable, denoted by X, lies in Euclidean space. We prove the almost sure convergence of the empirical version of these regions on the manifold to their population counterparts.
  arXiv  Detail & Related papers  (2023-10-12T10:56:25Z)
• Modeling the space-time correlation of pulsed twin beams [68.8204255655161]
  Entangled twin-beams generated by parametric down-conversion are among the favorite sources for imaging-oriented applications. We propose a semi-analytic model which aims to bridge the gap between time-consuming numerical simulations and the unrealistic plane-wave pump theory.
  arXiv  Detail & Related papers  (2023-01-18T11:29:49Z)
• Instance-Optimal Compressed Sensing via Posterior Sampling [101.43899352984774]
  We show for Gaussian measurements and emphany prior distribution on the signal, that the posterior sampling estimator achieves near-optimal recovery guarantees. We implement the posterior sampling estimator for deep generative priors using Langevin dynamics, and empirically find that it produces accurate estimates with more diversity than MAP.
  arXiv  Detail & Related papers  (2021-06-21T22:51:56Z)
• Loss function based second-order Jensen inequality and its application to particle variational inference [112.58907653042317]
  Particle variational inference (PVI) uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. We derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models.
  arXiv  Detail & Related papers  (2021-06-09T12:13:51Z)
• Non-asymptotic Optimal Prediction Error for Growing-dimensional Partially Functional Linear Models [0.951828574518325]
  We show the rate-optimal upper and lower bounds of the prediction error. An exact upper bound for the excess prediction risk is shown in a non-asymptotic form. We derive the non-asymptotic minimax lower bound under the regularity assumption of the Kullback-Leibler divergence of the models.
  arXiv  Detail & Related papers  (2020-09-10T08:49:32Z)
• Inference on the Change Point for High Dimensional Dynamic Graphical Models [9.74000189600846]
  We develop an estimator for the change point parameter for a dynamically evolving graphical model. It retains sufficient adaptivity against plug-in estimates of the graphical model parameters. It is illustrated on RNA-sequenced data and their changes between young and older individuals.
  arXiv  Detail & Related papers  (2020-05-19T19:15:32Z)
• Posterior Ratio Estimation of Latent Variables [14.619879849533662]
  In some applications, we want to compare distributions of random variables that are emphinferred from observations. We study the problem of estimating the ratio between two posterior probability density functions of a latent variable.
  arXiv  Detail & Related papers  (2020-02-15T16:46:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.