Bayesian Double Descent

• URL: http://arxiv.org/abs/2507.07338v1
• Date: Wed, 09 Jul 2025 23:47:26 GMT
• Title: Bayesian Double Descent
• Authors: Nick Polson, Vadim Sokolov,
• Abstract summary: We show a natural Bayesian interpretation of the double descent effect.<n>We show that it is not in conflict with the traditional Occam's razor that Bayesian models possess.<n>We illustrate the approach with an example of Bayesian model selection in neural networks.
• Score: 0.6906005491572398
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Double descent is a phenomenon of over-parameterized statistical models. Our goal is to view double descent from a Bayesian perspective. Over-parameterized models such as deep neural networks have an interesting re-descending property in their risk characteristics. This is a recent phenomenon in machine learning and has been the subject of many studies. As the complexity of the model increases, there is a U-shaped region corresponding to the traditional bias-variance trade-off, but then as the number of parameters equals the number of observations and the model becomes one of interpolation, the risk can become infinite and then, in the over-parameterized region, it re-descends -- the double descent effect. We show that this has a natural Bayesian interpretation. Moreover, we show that it is not in conflict with the traditional Occam's razor that Bayesian models possess, in that they tend to prefer simpler models when possible. We illustrate the approach with an example of Bayesian model selection in neural networks. Finally, we conclude with directions for future research.

Related papers

• Generative Modeling with Bayesian Sample Inference [50.07758840675341]
  We derive a novel generative model from iterative Gaussian posterior inference.<n>Our model uses a sequence of prediction and posterior update steps to iteratively narrow down the unknown sample.<n>In experiments, we demonstrate that our model improves sample quality on ImageNet32 over both BFNs and the closely related Variational Diffusion Models.
  arXiv  Detail & Related papers  (2025-02-11T14:27:10Z)
• Bayesian Circular Regression with von Mises Quasi-Processes [57.88921637944379]
  In this work we explore a family of expressive and interpretable distributions over circle-valued random functions.<n>For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Gibbs sampling.<n>We present experiments applying this model to the prediction of wind directions and the percentage of the running gait cycle as a function of joint angles.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Understanding the Double Descent Phenomenon in Deep Learning [49.1574468325115]
  This tutorial sets the classical statistical learning framework and introduces the double descent phenomenon. By looking at a number of examples, section 2 introduces inductive biases that appear to have a key role in double descent by selecting. section 3 explores the double descent with two linear models, and gives other points of view from recent related works.
  arXiv  Detail & Related papers  (2024-03-15T16:51:24Z)
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning [68.76846801719095]
  We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n. This provides a resolution to tensions between double descent and statistical intuition.
  arXiv  Detail & Related papers  (2023-10-29T12:05:39Z)
• Evidence Networks: simple losses for fast, amortized, neural Bayesian model comparison [0.0]
  Evidence Networks can enable Bayesian model comparison when state-of-the-art methods fail. We introduce the leaky parity-odd power transform, leading to the novel l-POP-Exponential'' loss function. We show that Evidence Networks are explicitly independent of dimensionality of the parameter space and scale mildly with the complexity of the posterior probability density function.
  arXiv  Detail & Related papers  (2023-05-18T18:14:53Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• Bayesian Neural Network Inference via Implicit Models and the Posterior Predictive Distribution [0.8122270502556371]
  We propose a novel approach to perform approximate Bayesian inference in complex models such as Bayesian neural networks. The approach is more scalable to large data than Markov Chain Monte Carlo. We see this being useful in applications such as surrogate and physics-based models.
  arXiv  Detail & Related papers  (2022-09-06T02:43:19Z)
• Multiple Descent in the Multiple Random Feature Model [8.988540634325691]
  We investigate the multiple descent phenomenon in a class of multi-component prediction models. We show that risk curves with a specific number of descent generally exist in learning multi-component prediction models.
  arXiv  Detail & Related papers  (2022-08-21T14:53:15Z)
• On the Role of Optimization in Double Descent: A Least Squares Study [30.44215064390409]
  We show an excess risk bound for the descent gradient solution of the least squares objective. We find that in case of noiseless regression, double descent is explained solely by optimization-related quantities. We empirically explore if our predictions hold for neural networks.
  arXiv  Detail & Related papers  (2021-07-27T09:13:11Z)
• A Bayesian Perspective on Training Speed and Model Selection [51.15664724311443]
  We show that a measure of a model's training speed can be used to estimate its marginal likelihood. We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks. Our results suggest a promising new direction towards explaining why neural networks trained with gradient descent are biased towards functions that generalize well.
  arXiv  Detail & Related papers  (2020-10-27T17:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.