OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework

• URL: http://arxiv.org/abs/2606.08574v1
• Date: Sun, 07 Jun 2026 11:11:51 GMT
• Title: OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Framework
• Abstract summary: OrderDP aims to obtain stable, unbiased, and near-lossless training acceleration with theoretical guarantees.<n>We evaluate OrderDP against comprehensive baselines on CIFAR-10, CIFAR-100, and ImageNet-1K.
• Score: 30.771327347534996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data pruning (DP), as an oft-stated strategy to alleviate heavy training burdens, reduces the volume of training samples according to a well-defined pruning method while striving for near-lossless performance. However, existing approaches, which commonly select highly informative samples, can lead to biased gradient estimation compared to full-dataset training. Furthermore, the analysis of this bias and its impact on final performance remains ambiguous. To address these challenges, we propose OrderDP, a plug-and-play framework that aims to obtain stable, unbiased, and near-lossless training acceleration with theoretical guarantees. Specifically, OrderDP first randomly selects a subset and then chooses the top-$q$ samples, where unbiasedness is established with respect to a surrogate loss. This ensures that OrderDP conducts unbiased training in terms of the surrogate objective. We further establish convergence and generalization analyses, elucidating how OrderDP affects optimal performance and enables well-controlled acceleration while ensuring guaranteed final performance. Empirically, we evaluate OrderDP against comprehensive baselines on CIFAR-10, CIFAR-100, and ImageNet-1K, demonstrating competitive accuracy, stable convergence, and exact control -- all with a simpler design and faster runtime, while reducing training cost by over 40%. Delivering both strong performance and computational efficiency, our method serves as a robust and easily adaptable tool for data-efficient learning. The code is publicly available at https://github.com/shengze-xu/OrderDP.

Related papers

• PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF [20.068789951726078]
  Reinforcement Learning from Human Feedback is a practical alternative to actor--critic training.<n>Existing critic-free approaches propagate a trajectory-level learning signal uniformly across all tokens in a trajectory.<n>We propose Prefix-Sampling Proximal Policy Optimization (PS-PPO), a compute-efficient critic-free method for RLHF.
  arXiv  Detail & Related papers  (2026-06-29T04:04:32Z)
• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration [71.77541419885925]
  We present a unified graph-based DP framework.<n>By modeling the dataset as a weighted graph, DP can be cast as a Maximum Weight Clique Problem (MWCP)<n>Our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
  arXiv  Detail & Related papers  (2026-06-11T05:13:32Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• GIPO: Gaussian Importance Sampling Policy Optimization [12.306486689840774]
  GIPO is proposed as a policy optimization objective based on truncated importance sampling.<n>It replaces hard clipping with a log-ratio-based Gaussian trust weight to damp extreme importance ratios.<n>GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes.
  arXiv  Detail & Related papers  (2026-03-04T11:34:59Z)
• Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards [26.5011687714416]
  Existing group-based policy optimization methods allocate a fixed number of rollouts for all training prompts.<n>This uniform allocation treats all prompts as equally informative, and could lead to inefficient computational budget usage and impede training progress.<n>We introduce VIP, a Variance-Informed Predictive allocation strategy that allocates a given rollout budget to the prompts in the incumbent batch to minimize the expected gradient variance of the policy update.
  arXiv  Detail & Related papers  (2026-02-02T03:50:01Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Scaling Adversarial Training via Data Selection [0.0]
  We propose emphSelective Adversa Training, which perturbs only a subset of critical samples in each minibatch.<n>Experiments on MNIST and CIFAR-10 show that the proposed methods achieve robustness comparable to, or even exceeding, full PGD adversarial training.
  arXiv  Detail & Related papers  (2025-12-26T15:50:33Z)
• Boosted Training of Lightweight Early Exits for Optimizing CNN Image Classification Inference [47.027290803102666]
  We introduce a sequential training approach that aligns branch training with inference-time data distributions.<n>Experiments on the CINIC-10 dataset with a ResNet18 backbone demonstrate that BTS-EE consistently outperforms non-boosted training.<n>These results offer practical efficiency gains for applications such as industrial inspection, embedded vision, and UAV-based monitoring.
  arXiv  Detail & Related papers  (2025-09-10T06:47:49Z)
• BAPE: Learning an Explicit Bayes Classifier for Long-tailed Visual Recognition [78.70453964041718]
  Current deep learning algorithms usually solve for the optimal classifier by emphimplicitly estimating the posterior probabilities.<n>This simple methodology has been proven effective for meticulously balanced academic benchmark datasets.<n>However, it is not applicable to the long-tailed data distributions in the real world.<n>This paper presents a novel approach (BAPE) that provides a more precise theoretical estimation of the data distributions.
  arXiv  Detail & Related papers  (2025-06-29T15:12:50Z)
• Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization [67.8738082040299]
  Self-Sampling Preference Optimization (SSPO) is a new alignment method for post-training reinforcement learning.<n>SSPO eliminates the need for paired data and reward models while retaining the training stability of SFT.<n>SSPO surpasses all previous approaches on the text-to-image benchmarks and demonstrates outstanding performance on the text-to-video benchmarks.
  arXiv  Detail & Related papers  (2024-10-07T17:56:53Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.