GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling

• URL: http://arxiv.org/abs/2508.19028v2
• Date: Wed, 27 Aug 2025 05:19:54 GMT
• Title: GRADSTOP: Early Stopping of Gradient Descent via Posterior Sampling
• Authors: Arash Jamshidi, Lauri Seppäläinen, Katsiaryna Haitsiukevich, Hoang Phuc Hau Luu, Anton Björklund, Kai Puolamäki,
• Abstract summary: Machine learning models often suffer from overfitting, leading to a decline in predictive performance on unseen data.<n>A standard solution is early stopping using a hold-out validation set, which halts the minimisation when the validation loss stops decreasing.<n>This paper presents GRADSTOP, a novel early stopping method that only uses information in the gradients, which are produced by the gradient descent algorithm.
• Score: 4.938367626424121
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Machine learning models are often learned by minimising a loss function on the training data using a gradient descent algorithm. These models often suffer from overfitting, leading to a decline in predictive performance on unseen data. A standard solution is early stopping using a hold-out validation set, which halts the minimisation when the validation loss stops decreasing. However, this hold-out set reduces the data available for training. This paper presents GRADSTOP, a novel stochastic early stopping method that only uses information in the gradients, which are produced by the gradient descent algorithm ``for free.'' Our main contributions are that we estimate the Bayesian posterior by the gradient information, define the early stopping problem as drawing sample from this posterior, and use the approximated posterior to obtain a stopping criterion. Our empirical evaluation shows that GRADSTOP achieves a small loss on test data and compares favourably to a validation-set-based stopping criterion. By leveraging the entire dataset for training, our method is particularly advantageous in data-limited settings, such as transfer learning. It can be incorporated as an optional feature in gradient descent libraries with only a small computational overhead. The source code is available at https://github.com/edahelsinki/gradstop.

Related papers

• Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting [15.251425165987987]
  Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities.<n>We propose a sample weighting scheme for the fine-tuning data based on the pre-trained model's losses.<n>We empirically demonstrate the efficacy of our method on both language and vision tasks.
  arXiv  Detail & Related papers  (2025-02-05T00:49:59Z)
• An Effective Dynamic Gradient Calibration Method for Continual Learning [11.555822066922508]
  Continual learning (CL) is a fundamental topic in machine learning, where the goal is to train a model with continuously incoming data and tasks. Due to the memory limit, we cannot store all the historical data, and therefore confront the catastrophic forgetting'' problem. We develop an effective algorithm to calibrate the gradient in each updating step of the model.
  arXiv  Detail & Related papers  (2024-07-30T16:30:09Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [93.90047628101155]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.<n>To address this, some methods propose replaying data from previous tasks during new task learning.<n>However, it is not expected in practice due to memory constraints and data privacy issues.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• A Negative Result on Gradient Matching for Selective Backprop [8.463693396893731]
  Training deep neural networks becomes a massive computational burden. One approach to speed up the training process is Selective Backprop. We build on this approach by choosing the (weighted) subset which best matches the mean gradient over the entire minibatch. We find that both the loss-based as well as the gradient-matching strategy fail to consistently outperform the random baseline.
  arXiv  Detail & Related papers  (2023-12-08T13:03:10Z)
• Dropout Reduces Underfitting [85.61466286688385]
  In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training. We find dropout reduces the directional variance of gradients across mini-batches and helps align the mini-batch gradients with the entire dataset's gradient. Our findings lead us to a solution for improving performance in underfitting models - early dropout: dropout is applied only during the initial phases of training, and turned off afterwards.
  arXiv  Detail & Related papers  (2023-03-02T18:59:15Z)
• Active Labeling: Streaming Stochastic Gradients [91.76135191049232]
  We formalize the "active labeling" problem, which generalizes active learning based on partial supervision. We provide a streaming technique that minimizes the ratio of generalization error over number of samples.
  arXiv  Detail & Related papers  (2022-05-26T09:49:16Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• Carath\'eodory Sampling for Stochastic Gradient Descent [79.55586575988292]
  We present an approach that is inspired by classical results of Tchakaloff and Carath'eodory about measure reduction. We adaptively select the descent steps where the measure reduction is carried out. We combine this with Block Coordinate Descent so that measure reduction can be done very cheaply.
  arXiv  Detail & Related papers  (2020-06-02T17:52:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.