Understanding the Training Speedup from Sampling with Approximate Losses

• URL: http://arxiv.org/abs/2402.07052v2
• Date: Wed, 04 Jun 2025 03:06:13 GMT
• Title: Understanding the Training Speedup from Sampling with Approximate Losses
• Abstract summary: In this work, we focus on the greedy approach of selecting samples with large textitapproximate losses instead of exact losses.<n>For smooth convex losses, we show that such a greedy strategy can converge to a constant factor of the minimum value of the average loss in fewer iterations.<n>We then develop SIFT which uses early exiting to obtain approximate losses with an intermediate layer's representations for sample selection.
• Score: 14.166048343288157
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yield any meaningful gains in terms of overall training time. In this work, we focus on the greedy approach of selecting samples with large \textit{approximate losses} instead of exact losses in order to reduce the selection overhead. For smooth convex losses, we show that such a greedy strategy can converge to a constant factor of the minimum value of the average loss in fewer iterations than the standard approach of random selection. We also theoretically quantify the effect of the approximation level. We then develop SIFT which uses early exiting to obtain approximate losses with an intermediate layer's representations for sample selection. We evaluate SIFT on the task of training a 110M parameter 12 layer BERT base model, and show significant gains (in terms of training hours and number of backpropagation steps) without any optimized implementation over vanilla training. For e.g., to reach 64% validation accuracy, SIFT with exit at the first layer takes ~ 43 hours compared to ~ 57 hours of vanilla training.

Related papers

• Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference [15.267768829315408]
  Layer dropout has been shown to enable faster training and higher accuracy.<n> Layer dropout has largely disappeared from large language models pre-training recipes.<n>We show that layer dropout should be used in state-of-the-art LLM training.
  arXiv  Detail & Related papers  (2026-09-04T15:30:47Z)
• Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization [0.33842793760651557]
  We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique.<n>PBS prioritizes high-loss samples during batch construction.<n>Our predictor achieves 0.44 correlation with actual loss using only four simple features.
  arXiv  Detail & Related papers  (2026-02-19T04:15:39Z)
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling [75.36692892951018]
  Increasing the batch size during training is a promising strategy to accelerate large language model pretraining.<n>This work develops a principled framework for batch-size scheduling.<n>It introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/sqrt2$ and doubles the batch size.
  arXiv  Detail & Related papers  (2025-10-16T14:17:38Z)
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining [53.893792844055106]
  Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency.<n>We introduce Selective Efficient Language Modeling, a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection.<n> Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines.
  arXiv  Detail & Related papers  (2025-05-26T12:23:26Z)
• SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement [100.85923086072204]
  ThinkLite-VL improves the average performance of Qwen2.5-VL-7B-Instruct by 7%, using only 11k training samples with no knowledge distillation. Our code, data, and model are available at https://github.com/si0wang/ThinkLite-VL.
  arXiv  Detail & Related papers  (2025-04-10T17:49:05Z)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
  We present the first systematic exploration of optimal sparse pre-training configurations for large language models. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
  arXiv  Detail & Related papers  (2025-01-21T20:23:22Z)
• Efficient Neural Network Training via Subset Pretraining [5.352839075466439]
  In training neural networks, it is common practice to use partial gradients computed over batches. The loss minimum of the training set can be expected to be well-approximated by the minima of its subsets. experiments have confirmed that results equivalent to conventional training can be reached.
  arXiv  Detail & Related papers  (2024-10-21T21:31:12Z)
• Robust Time Series Forecasting with Non-Heavy-Tailed Gaussian Loss-Weighted Sampler [1.8816077341295625]
  Recent resampling methods aim to increase training efficiency by reweighting samples based on their running losses. We introduce a novel approach: a Gaussian loss-weighted sampler that multiplies their running losses with a Gaussian distribution weight. It reduces the probability of selecting samples with very low or very high losses while favoring those close to average losses.
  arXiv  Detail & Related papers  (2024-06-19T22:28:18Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• A Negative Result on Gradient Matching for Selective Backprop [8.463693396893731]
  Training deep neural networks becomes a massive computational burden. One approach to speed up the training process is Selective Backprop. We build on this approach by choosing the (weighted) subset which best matches the mean gradient over the entire minibatch. We find that both the loss-based as well as the gradient-matching strategy fail to consistently outperform the random baseline.
  arXiv  Detail & Related papers  (2023-12-08T13:03:10Z)
• Data Pruning via Moving-one-Sample-out [61.45441981346064]
  We propose a novel data-pruning approach called moving-one-sample-out (MoSo) MoSo aims to identify and remove the least informative samples from the training set. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios.
  arXiv  Detail & Related papers  (2023-10-23T08:00:03Z)
• KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training [2.8804804517897935]
  We propose a method for hiding the least-important samples during the training of deep neural networks. We adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process. Our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline.
  arXiv  Detail & Related papers  (2023-10-16T06:19:29Z)
• Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks [70.75043144299168]
  Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. We propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency. Our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.
  arXiv  Detail & Related papers  (2023-02-28T05:01:01Z)
• Prioritizing Samples in Reinforcement Learning with Reducible Loss [5.901819658403315]
  We propose a method to prioritize samples based on how much we can learn from a sample. We develop an algorithm to prioritize samples with high learn-ability, while assigning lower priority to those that are hard-to-learn.
  arXiv  Detail & Related papers  (2022-08-22T17:55:43Z)
• Mixing between the Cross Entropy and the Expectation Loss Terms [89.30385901335323]
  Cross entropy loss tends to focus on hard to classify samples during training. We show that adding to the optimization goal the expectation loss helps the network to achieve better accuracy. Our experiments show that the new training protocol improves performance across a diverse set of classification domains.
  arXiv  Detail & Related papers  (2021-09-12T23:14:06Z)
• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels [145.06552420999986]
  In learning with noisy labels, the sample selection approach is very popular, which regards small-loss data as correctly labeled during training. However, losses are generated on-the-fly based on the model being trained with noisy labels, and thus large-loss data are likely but not certainly to be incorrect. In this paper, we incorporate the uncertainty of losses by adopting interval estimation instead of point estimation of losses.
  arXiv  Detail & Related papers  (2021-06-01T12:53:53Z)
• Predicting Training Time Without Training [120.92623395389255]
  We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
  arXiv  Detail & Related papers  (2020-08-28T04:29:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.