Batch Loss Score for Dynamic Data Pruning

• URL: http://arxiv.org/abs/2604.04681v1
• Date: Mon, 06 Apr 2026 13:48:11 GMT
• Title: Batch Loss Score for Dynamic Data Pruning
• Abstract summary: Batch Loss Score (BLS) is a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples.<n>BLS approximats the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance.
• Score: 30.39085035238577
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Dynamic data pruning accelerates deep learning by selectively omitting less informative samples during training. While per-sample loss is a common importance metric, obtaining it can be challenging or infeasible for complex models or loss functions, often requiring significant implementation effort. This work proposes the Batch Loss Score (BLS), a computationally efficient alternative using an Exponential Moving Average (EMA) of readily available batch losses to assign scores to individual samples. We frame the batch loss, from the perspective of a single sample, as a noisy measurement of its scaled individual loss, with noise originating from stochastic batch composition. It is formally shown that the EMA mechanism functions as a first-order low-pass filter, attenuating high-frequency batch composition noise. This yields a score approximating the smoothed and persistent contribution of the individual sample to the loss, providing a theoretical grounding for BLS as a proxy for sample importance. BLS demonstrates remarkable code integration simplicity (\textbf{three-line injection}) and readily adapts existing per-sample loss-based methods (\textbf{one-line proxy}). Its effectiveness is demonstrated by enhancing two such methods to losslessly prune \textbf{20\%-50\%} of samples across \textit{14 datasets}, \textit{11 tasks} and \textit{18 models}, highlighting its utility and broad applicability, especially for complex scenarios where per-sample loss is difficult to access. Code is available at https://github.com/mrazhou/BLS.

Related papers

• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)
• Enhancing Sample Utilization in Noise-Robust Deep Metric Learning With Subgroup-Based Positive-Pair Selection [84.78475642696137]
  The existence of noisy labels in real-world data negatively impacts the performance of deep learning models.<n>We propose a noise-robust DML framework with SubGroup-based Positive-pair Selection (SGPS)<n>SGPS constructs reliable positive pairs for noisy samples to enhance the sample utilization.
  arXiv  Detail & Related papers  (2025-01-19T14:41:55Z)
• Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler [1.8816077341295625]
  Recent resampling methods aim to increase training efficiency by reweighting samples based on their running losses. We introduce a novel approach: a Gaussian loss-weighted sampler that multiplies their running losses with a Gaussian distribution weight. It reduces the probability of selecting samples with very low or very high losses while favoring those close to average losses.
  arXiv  Detail & Related papers  (2024-06-19T22:28:18Z)
• Double Correction Framework for Denoising Recommendation [45.98207284259792]
  In implicit feedback, noisy samples can affect precise user preference learning. A popular solution is based on dropping noisy samples in the model training phase. We propose a Double Correction Framework for Denoising Recommendation.
  arXiv  Detail & Related papers  (2024-05-18T12:15:10Z)
• PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learning [7.556169113399857]
  We propose an end-to-end textbfPLReMix framework by introducing a Pseudo-Label Relaxed (PLR) contrastive loss.<n>The proposed PLR loss is pluggable and we have integrated it into other LNL methods, observing their improved performance.
  arXiv  Detail & Related papers  (2024-02-27T15:22:20Z)
• Sample Weight Estimation Using Meta-Updates for Online Continual Learning [7.832189413179361]
  Online Meta-learning for Sample Importance (OMSI) strategy approximates sample weights for a mini-batch in an online CL stream. OMSI enhances both learning and retained accuracy in a controlled noisy-labeled data stream.
  arXiv  Detail & Related papers  (2024-01-29T09:04:45Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Regroup Median Loss for Combating Label Noise [19.51996047333779]
  Deep model training requires large-scale datasets of annotated data. Due to the difficulty of annotating a large number of samples, label noise caused by incorrect annotations is inevitable. We propose Regroup Median Loss (RML) to reduce the probability of selecting noisy samples and correct losses of noisy samples.
  arXiv  Detail & Related papers  (2023-12-11T10:19:55Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)
• Salvage Reusable Samples from Noisy Data for Robust Learning [70.48919625304]
  We propose a reusable sample selection and correction approach, termed as CRSSC, for coping with label noise in training deep FG models with web images. Our key idea is to additionally identify and correct reusable samples, and then leverage them together with clean examples to update the networks.
  arXiv  Detail & Related papers  (2020-08-06T02:07:21Z)
• Robust and On-the-fly Dataset Denoising for Image Classification [72.10311040730815]
  On-the-fly Data Denoising (ODD) is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
  arXiv  Detail & Related papers  (2020-03-24T03:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.