Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training

• URL: http://arxiv.org/abs/2605.14773v1
• Date: Thu, 14 May 2026 12:37:11 GMT
• Title: Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training
• Abstract summary: We show that selected-data training induces an implicit regularization effect by the instantaneous selection ratio.<n>This reveals a key trade-off: lower amplify selection-induced regularization, whereas higher ratios preserve data coverage.<n> PODS is compatible with existing static and dynamic selection methods and broadly applicable across training paradigms.
• Score: 36.289092974454796
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data selection accelerates training by identifying representative training data while preserving model performance. However, existing methods mainly focus on designing sample-importance criteria, i.e., deciding what to select, while typically fixing the selected data volume as the target ratio throughout training. Thus, they are often dynamic in sample identity but static in data volume. In this work, we revisit data selection from an optimization perspective and show that selected-data training induces an implicit regularization effect modulated by the instantaneous selection ratio. This reveals a key trade-off: lower ratios amplify selection-induced regularization, whereas higher ratios preserve data coverage and optimization fidelity. Motivated by this insight, we propose PODS, a Plug-and-play Oscillatory Data-volume Scheduling framework. Rather than introducing another sample-scoring metric, PODS serves as a lightweight module that dynamically schedules how much data to select over training. Under the target selection ratio, PODS alternates between low-ratio regularization phases and high-ratio recovery phases to exploit selection-induced regularization without sacrificing optimization stability. With its lightweight, ratio-level, and task-agnostic design, PODS is compatible with existing static and dynamic selection methods and broadly applicable across training paradigms. Experiments across various datasets, architectures, and tasks show that PODS consistently improves the efficiency-generalization trade-off, e.g., reducing ImageNet-1k training cost by 50% with improved accuracy and accelerating LLM instruction tuning by over 2x without performance degradation.

Related papers

• Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training [9.040054119267344]
  Dr. Post-Training is a novel framework that reconceptualizes general training data as a data-induced regularizer.<n>We show that our methods consistently outperform state-of-the-art data selection baselines.
  arXiv  Detail & Related papers  (2026-05-08T00:16:11Z)
• EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation [79.71802168256542]
  Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.<n>One straightforward approach is through an iterative generation-training loop, where candidate data are synthesized through an external generator.<n>We introduce a refined paradigm, namely an iterative generation-selection-training loop, which incorporates a selection step prior to model updates.
  arXiv  Detail & Related papers  (2026-04-28T23:26:16Z)
• Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks [10.269961654607108]
  Deep neural networks are typically trained by uniformly sampling large datasets across epochs.<n>Recent work shows that progressively reducing the amount of training data can improve efficiency and generalization.<n>We propose Adaptive Data Dropout, a framework that dynamically adjusts the subset of training data based on performance feedback.
  arXiv  Detail & Related papers  (2026-04-14T16:41:33Z)
• Data Agent: Learning to Select Data via End-to-End Dynamic Optimization [37.1771265765151]
  Data Agent formulates data selection as a training-aware sequential decision-making problem.<n>Experiments demonstrate that Data Agent consistently accelerates training while preserving or improving performance.<n>Its dataset-agnostic formulation and modular reward make it plug-and-play across tasks and scenarios.
  arXiv  Detail & Related papers  (2026-03-08T03:10:39Z)
• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning [49.04912820721943]
  Supervised fine-tuning (SFT) is computationally expensive and sometimes suffers from overfitting or bias amplification.<n>This work studies the online batch selection family that dynamically scores and filters samples during the training process.<n>We develop textbfUDS (Utility-Diversity Sampling), a framework for efficient online batch selection in SFT.
  arXiv  Detail & Related papers  (2025-10-19T15:32:01Z)
• LLM Data Selection and Utilization via Dynamic Bi-level Optimization [100.20933466418786]
  We propose a new Data Weighting Model (DWM) to adjust the weight of selected data within each batch to achieve a dynamic data utilization during training.<n>Our experiments demonstrate that DWM enhances the performance of models trained with randomly-selected data.<n>We further analyze how a model's data preferences evolve throughout training, providing new insights into the data preference of the model during training.
  arXiv  Detail & Related papers  (2025-07-22T02:47:12Z)
• RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessment [10.284993431741377]
  We introduce the concept of epsilon-sample cover, which quantifies sample redundancy based on inter-sample relationships.<n>We reformulate data selection as a reinforcement learning process and propose RL-Selector.<n>Our method consistently outperforms existing state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-06-26T06:28:56Z)
• OASIS: Online Sample Selection for Continual Visual Instruction Tuning [55.92362550389058]
  In continual instruction tuning (CIT) scenarios, new instruction tuning data continuously arrive in an online streaming manner.<n>Data selection can mitigate this overhead, but existing strategies often rely on pretrained reference models.<n>Recent reference model-free online sample selection methods address this, but typically select a fixed number of samples per batch.
  arXiv  Detail & Related papers  (2025-05-27T20:32:43Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.<n>Data selection has shown promise in identifying the most representative samples from the entire dataset.<n>We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning [68.12870241637636]
  We propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training. Our empirical results indicate that MILO can train models $3times - 10 times$ faster and tune hyperparameters $20times - 75 times$ faster than full-dataset training or tuning without performance.
  arXiv  Detail & Related papers  (2023-01-30T20:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.