Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization

• URL: http://arxiv.org/abs/2602.17066v1
• Date: Thu, 19 Feb 2026 04:15:39 GMT
• Title: Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization
• Abstract summary: We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique.<n>PBS prioritizes high-loss samples during batch construction.<n>Our predictor achieves 0.44 correlation with actual loss using only four simple features.
• Score: 0.33842793760651557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique that accelerates language model convergence by dynamically prioritizing high-loss samples during batch construction. Unlike curriculum learning approaches that require predefined difficulty metrics or hard example mining methods that demand expensive per-sample loss tracking, PBS employs a lightweight linear predictor trained online to estimate sample difficulty from static token-level features. Our predictor achieves 0.44 correlation with actual loss using only four simple features: token frequency, sequence length, vocabulary diversity, and rare token ratio. Experiments on a 130M parameter transformer demonstrate that PBS achieves 6-13\% faster convergence measured by evaluation loss across training checkpoints, with the predictor's correlation improving from 0.14 to 0.44 over 10,000 training steps. These results validate that token frequency statistics encode meaningful information about sample difficulty, enabling effective curriculum learning with negligible computational overhead.

Related papers

• A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget [0.0]
  This study examines training dynamics in a small Llama-style language model trained under a compute-constrained token budget.<n>Six independent training runs were conducted using the TinyStories corpus, CPU-based full-precision training, and a target budget of approximately 20 million cumulative training tokens.<n>Repeated measures ANOVA showed statistically significant interval effects for validation loss, validation perplexity, and rolling volatility.
  arXiv  Detail & Related papers  (2026-06-11T13:55:26Z)
• Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification [14.726932796075927]
  TRIAGE is a tiered zero-shot framework that scales test-time compute by routing each audio sample through richer reasoning stages.<n>A confidence-based router finalizes easy predictions early while allocating additional computation to ambiguous inputs.<n>Across nine respiratory classification tasks without task-specific training, TRIAGE achieves a mean AUROC of 0.744, outperforming prior zero-shot methods.
  arXiv  Detail & Related papers  (2026-04-14T12:17:50Z)
• Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning [43.11305591635628]
  Olmo3-7B trained for 128 epochs on 400 samples outperforms the equivalent 1 epoch on 51200 samples by 12-26 percentage points.<n>We find that training token accuracy reliably signals when repetition has saturated.
  arXiv  Detail & Related papers  (2026-02-11T18:58:54Z)
• Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training [11.179110411255708]
  We propose a direct framework to model the scaling of benchmark performance from the training budget.<n>Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure.<n>We release the complete set of pretraining losses and downstream evaluation results.
  arXiv  Detail & Related papers  (2025-12-09T18:33:48Z)
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)
• Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding [53.63482987410292]
  We present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models.<n>We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks.
  arXiv  Detail & Related papers  (2025-07-13T19:36:17Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models [15.50128790503447]
  We propose a novel and theoretically motivated methodology for pre-training data detection, named Min-K%++.<n>Specifically, we present a key insight that training samples tend to be local maxima of the modeled distribution along each input dimension through likelihood training.
  arXiv  Detail & Related papers  (2024-04-03T04:25:01Z)
• Understanding the Training Speedup from Sampling with Approximate Losses [14.166048343288157]
  In this work, we focus on the greedy approach of selecting samples with large textitapproximate losses instead of exact losses.<n>For smooth convex losses, we show that such a greedy strategy can converge to a constant factor of the minimum value of the average loss in fewer iterations.<n>We then develop SIFT which uses early exiting to obtain approximate losses with an intermediate layer's representations for sample selection.
  arXiv  Detail & Related papers  (2024-02-10T21:51:59Z)
• Revisiting the Power of Prompt for Visual Tuning [50.11465784194896]
  This study explores the correlation evolvement between prompts and patch tokens during proficient training. Inspired by the observation that the prompt tokens tend to share high mutual information with patch tokens, we propose initializing prompts with downstream token prototypes. Our method significantly advances the adaptation for self-supervised pretraining, achieving impressive task performance gains of at least 10% to 30%.
  arXiv  Detail & Related papers  (2024-02-04T07:49:02Z)
• Online Importance Sampling for Stochastic Gradient Optimization [33.42221341526944]
  We propose a practical algorithm that efficiently computes data importance on-the-fly during training.<n>We also introduce a novel metric based on the derivative of the loss w.r.t. the network output, designed for mini-batch importance sampling.
  arXiv  Detail & Related papers  (2023-11-24T13:21:35Z)
• KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training [2.8804804517897935]
  We propose a method for hiding the least-important samples during the training of deep neural networks. We adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process. Our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline.
  arXiv  Detail & Related papers  (2023-10-16T06:19:29Z)
• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction [55.77136037458667]
  We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization. We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
  arXiv  Detail & Related papers  (2023-04-20T07:29:23Z)
• Robust Few-shot Learning Without Using any Adversarial Samples [19.34427461937382]
  A few efforts have been made to combine the few-shot problem with the robustness objective using sophisticated Meta-Learning techniques. We propose a simple but effective alternative that does not require any adversarial samples. Inspired by the cognitive decision-making process in humans, we enforce high-level feature matching between the base class data and their corresponding low-frequency samples.
  arXiv  Detail & Related papers  (2022-11-03T05:58:26Z)
• Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models [107.05966685291067]
  We propose test-time prompt tuning (TPT) to learn adaptive prompts on the fly with a single test sample. TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data.
  arXiv  Detail & Related papers  (2022-09-15T17:55:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.