Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Information

• URL: http://arxiv.org/abs/2507.00038v2
• Date: Mon, 14 Jul 2025 09:02:45 GMT
• Title: Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Information
• Authors: Fei Chen, Wenchi Zhou,
• Abstract summary: We propose an effective data reduction strategy based on Pointwise - Information (PVI)<n>Experiments show that the classifier performance is maintained with only a 0.0001% to 0.76% reduction in accuracy when 10%-30% of the data is removed.<n>We have adapted the PVI framework, which was previously limited to English datasets, to a variety of Chinese NLP tasks and base models.
• Score: 2.133855532092057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In order to increase the effectiveness of model training, data reduction is essential to data-centric AI. It does this by locating the most instructive examples in massive datasets. To increase data quality and training efficiency, the main difficulty is to choose the best examples rather than the complete datasets. In this paper, we propose an effective data reduction strategy based on Pointwise -Information (PVI). To enable a static method, we first use PVI to quantify instance difficulty and remove instances with low difficulty. Experiments show that the classifier performance is maintained with only a 0.0001% to 0.76% reduction in accuracy when 10%-30% of the data is removed. Second, we train the classifiers using a progressive learning strategy on examples sorted by increasing PVI, accelerating convergence and achieving a 0.8% accuracy gain over conventional training. Our findings imply that training a classifier on the chosen optimal subset may improve model performance and increase training efficiency when combined with an efficient data reduction strategy. Furthermore, we have adapted the PVI framework, which was previously limited to English datasets, to a variety of Chinese NLP tasks and base models, yielding insightful results for faster training and cross-lingual data reduction. The codes are released at https://github.com/zhouwenchi/DatasetReductionStrategy.

Related papers

• LearnAlign: Reasoning Data Selection for Reinforcement Learning in Large Language Models Based on Improved Gradient Alignment [14.655048266761783]
  Reinforcement learning (RL) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck.<n>We present LearnAlign, which intelligently selects the learnable and representative training reasoning data for RL post-training.<n> Experiments across three mathematical reasoning benchmarks demonstrate that our method significantly reduces training data requirements.
  arXiv  Detail & Related papers  (2025-06-13T06:05:58Z)
• LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning [22.242445543184264]
  We propose LEAD, an efficient iterative data selection framework that accurately estimates sample utility entirely within the standard training loop.<n>Experiments show that LEAD significantly outperforms state-of-the-art methods, improving average model performance by 6.1%-10.8% while using only 2.5% of the training data and reducing overall training time by 5-10x.
  arXiv  Detail & Related papers  (2025-05-12T10:57:51Z)
• Data-efficient LLM Fine-tuning for Code Generation [30.441431693349866]
  Large language models (LLMs) have demonstrated significant potential in code generation tasks.<n>Existing approaches typically generate large amounts of synthetic data for fine-tuning.<n>We propose a data selection strategy in order to improve the effectiveness and efficiency of training for code-based LLMs.
  arXiv  Detail & Related papers  (2025-04-17T06:29:28Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.<n>Data selection has shown promise in identifying the most representative samples from the entire dataset.<n>We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• Exploring Learning Complexity for Efficient Downstream Dataset Pruning [8.990878450631596]
  Existing dataset pruning methods require training on the entire dataset.<n>We propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC)<n>Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
  arXiv  Detail & Related papers  (2024-02-08T02:29:33Z)
• Effective pruning of web-scale datasets based on complexity of concept clusters [48.125618324485195]
  We present a method for pruning large-scale multimodal datasets for training CLIP-style models on ImageNet. We find that training on a smaller set of high-quality data can lead to higher performance with significantly lower training costs. We achieve a new state-of-the-art Imagehttps://info.arxiv.org/help/prep#commentsNet zero-shot accuracy and a competitive average zero-shot accuracy on 38 evaluation tasks.
  arXiv  Detail & Related papers  (2024-01-09T14:32:24Z)
• Online Importance Sampling for Stochastic Gradient Optimization [33.42221341526944]
  We propose a practical algorithm that efficiently computes data importance on-the-fly during training.<n>We also introduce a novel metric based on the derivative of the loss w.r.t. the network output, designed for mini-batch importance sampling.
  arXiv  Detail & Related papers  (2023-11-24T13:21:35Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data [122.282521548393]
  Contrastive Language-Image Pre-training (CLIP) has become the standard for cross-modal image-text representation learning.<n>We introduce HELIP, a cost-effective strategy that improves CLIP models by exploiting challenging text-image pairs within existing datasets in continuous training.
  arXiv  Detail & Related papers  (2023-05-09T07:00:17Z)
• Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information [100.03188187735624]
  We introduce a novel approach based on PLMs and pointwise V-information (PVI), a metric that can measure the usefulness of a datapoint for training a model. Our method first fine-tunes a PLM on a small seed of training data and then synthesizes new datapoints - utterances that correspond to given intents. Our method is thus able to leverage the expressive power of large language models to produce diverse training data.
  arXiv  Detail & Related papers  (2023-02-10T07:37:49Z)
• Efficient Conditional Pre-training for Transfer Learning [71.01129334495553]
  We propose efficient filtering methods to select relevant subsets from the pre-training dataset. We validate our techniques by pre-training on ImageNet in both the unsupervised and supervised settings. We improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.
  arXiv  Detail & Related papers  (2020-11-20T06:16:15Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.