The Double Descent Behavior in Two Layer Neural Network for Binary Classification

• URL: http://arxiv.org/abs/2504.19351v1
• Date: Sun, 27 Apr 2025 20:29:24 GMT
• Title: The Double Descent Behavior in Two Layer Neural Network for Binary Classification
• Authors: Chathurika S Abeykoon, Aleksandr Beknazaryan, Hailin Sang,
• Abstract summary: Recent studies observed a surprising concept on model test error called the double descent phenomenon.<n>Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes.
• Score: 46.3107850275261
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.

Related papers

• Supervised Score-Based Modeling by Gradient Boosting [49.556736252628745]
  We propose a Supervised Score-based Model (SSM) which can be viewed as a gradient boosting algorithm combining score matching.<n>We provide a theoretical analysis of learning and sampling for SSM to balance inference time and prediction accuracy.<n>Our model outperforms existing models in both accuracy and inference time.
  arXiv  Detail & Related papers  (2024-11-02T07:06:53Z)
• Understanding the Double Descent Phenomenon in Deep Learning [49.1574468325115]
  This tutorial sets the classical statistical learning framework and introduces the double descent phenomenon. By looking at a number of examples, section 2 introduces inductive biases that appear to have a key role in double descent by selecting. section 3 explores the double descent with two linear models, and gives other points of view from recent related works.
  arXiv  Detail & Related papers  (2024-03-15T16:51:24Z)
• The twin peaks of learning neural networks [3.382017614888546]
  Recent works demonstrated the existence of a double-descent phenomenon for the generalization error of neural networks. We explore a link between this phenomenon and the increase of complexity and sensitivity of the function represented by neural networks.
  arXiv  Detail & Related papers  (2024-01-23T10:09:14Z)
• Analysis of Interpolating Regression Models and the Double Descent Phenomenon [3.883460584034765]
  It is commonly assumed that models which interpolate noisy training data are poor to generalize. The best models obtained are overparametrized and the testing error exhibits the double descent behavior as the model order increases. We derive a result based on the behavior of the smallest singular value of the regression matrix that explains the peak location and the double descent shape of the testing error as a function of model order.
  arXiv  Detail & Related papers  (2023-04-17T09:44:33Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• On the Role of Optimization in Double Descent: A Least Squares Study [30.44215064390409]
  We show an excess risk bound for the descent gradient solution of the least squares objective. We find that in case of noiseless regression, double descent is explained solely by optimization-related quantities. We empirically explore if our predictions hold for neural networks.
  arXiv  Detail & Related papers  (2021-07-27T09:13:11Z)
• Optimization Variance: Exploring Generalization Properties of DNNs [83.78477167211315]
  The test error of a deep neural network (DNN) often demonstrates double descent. We propose a novel metric, optimization variance (OV), to measure the diversity of model updates.
  arXiv  Detail & Related papers  (2021-06-03T09:34:17Z)
• A Bayesian Perspective on Training Speed and Model Selection [51.15664724311443]
  We show that a measure of a model's training speed can be used to estimate its marginal likelihood. We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks. Our results suggest a promising new direction towards explaining why neural networks trained with gradient descent are biased towards functions that generalize well.
  arXiv  Detail & Related papers  (2020-10-27T17:56:14Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Optimal Regularization Can Mitigate Double Descent [29.414119906479954]
  We study whether the double-descent phenomenon can be avoided by using optimal regularization. We demonstrate empirically that optimally-tuned $ell$ regularization can double descent for more general models, including neural networks.
  arXiv  Detail & Related papers  (2020-03-04T05:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.