Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty

• URL: http://arxiv.org/abs/2502.06905v1
• Date: Mon, 10 Feb 2025 01:18:40 GMT
• Title: Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
• Authors: Yeseul Cho, Baekrok Shin, Changmin Kang, Chulhee Yun,
• Abstract summary: We introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score to identify important samples from the early training stage. We also propose a ratio-adaptive sampling using Beta distribution to address a catastrophic accuracy drop at an extreme pruning.
• Score: 9.749638953163391
• License:
• Abstract: Recent advances in deep learning rely heavily on massive datasets, leading to substantial storage and training costs. Dataset pruning aims to alleviate this demand by discarding redundant examples. However, many existing methods require training a model with a full dataset over a large number of epochs before being able to prune the dataset, which ironically makes the pruning process more expensive than just training the model on the entire dataset. To overcome this limitation, we introduce a Difficulty and Uncertainty-Aware Lightweight (DUAL) score, which aims to identify important samples from the early training stage by considering both example difficulty and prediction uncertainty. To address a catastrophic accuracy drop at an extreme pruning, we further propose a ratio-adaptive sampling using Beta distribution. Experiments on various datasets and learning scenarios such as image classification with label noise and image corruption, and model architecture generalization demonstrate the superiority of our method over previous state-of-the-art (SOTA) approaches. Specifically, on ImageNet-1k, our method reduces the time cost for pruning to 66% compared to previous methods while achieving a SOTA, specifically 60% test accuracy at a 90% pruning ratio. On CIFAR datasets, the time cost is reduced to just 15% while maintaining SOTA performance.

Related papers

• Data Pruning Can Do More: A Comprehensive Data Pruning Approach for Object Re-identification [13.732596789612362]
  This work is the first to explore the feasibility of data pruning methods applied to object re-identification tasks. By fully leveraging the logit history during training, our approach offers a more accurate and comprehensive metric for quantifying sample importance. Our approach is highly efficient, reducing the cost of importance score estimation by 10 times compared to existing methods.
  arXiv  Detail & Related papers  (2024-12-13T12:27:47Z)
• PUMA: margin-based data pruning [51.12154122266251]
  We focus on data pruning, where some training samples are removed based on the distance to the model classification boundary (i.e., margin) We propose PUMA, a new data pruning strategy that computes the margin using DeepFool. We show that PUMA can be used on top of the current state-of-the-art methodology in robustness, and it is able to significantly improve the model performance unlike the existing data pruning strategies.
  arXiv  Detail & Related papers  (2024-05-10T08:02:20Z)
• A Study in Dataset Pruning for Image Super-Resolution [9.512648704408095]
  We introduce a novel approach that reduces a dataset to a core-set of training samples, selected based on their loss values. We achieve results comparable to or surpassing those obtained from training on the entire dataset.
  arXiv  Detail & Related papers  (2024-03-25T18:16:34Z)
• Exploring Learning Complexity for Efficient Downstream Dataset Pruning [8.990878450631596]
  Existing dataset pruning methods require training on the entire dataset. We propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC) Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
  arXiv  Detail & Related papers  (2024-02-08T02:29:33Z)
• Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning [50.809769498312434]
  We propose a novel dataset pruning method termed as Temporal Dual-Depth Scoring (TDDS) Our method achieves 54.51% accuracy with only 10% training data, surpassing random selection by 7.83% and other comparison methods by at least 12.69%.
  arXiv  Detail & Related papers  (2023-11-22T03:45:30Z)
• KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training [2.8804804517897935]
  We propose a method for hiding the least-important samples during the training of deep neural networks. We adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process. Our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline.
  arXiv  Detail & Related papers  (2023-10-16T06:19:29Z)
• Large-scale Dataset Pruning with Dynamic Uncertainty [28.60845105174658]
  The state of the art of many learning tasks, e.g., image classification, is advanced by collecting larger datasets and then training larger models on them. In this paper, we investigate how to prune the large-scale datasets, and thus produce an informative subset for training sophisticated deep models with negligible performance drop.
  arXiv  Detail & Related papers  (2023-06-08T13:14:35Z)
• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction [55.77136037458667]
  We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization. We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
  arXiv  Detail & Related papers  (2023-04-20T07:29:23Z)
• Gradient-Free Structured Pruning with Unlabeled Data [57.999191898036706]
  We propose a gradient-free structured pruning framework that uses only unlabeled data. Up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
  arXiv  Detail & Related papers  (2023-03-07T19:12:31Z)
• On Measuring the Intrinsic Few-Shot Hardness of Datasets [49.37562545777455]
  We show that few-shot hardness may be intrinsic to datasets, for a given pre-trained model. We propose a simple and lightweight metric called "Spread" that captures the intuition that few-shot learning is made possible. Our metric better accounts for few-shot hardness compared to existing notions of hardness, and is 8-100x faster to compute.
  arXiv  Detail & Related papers  (2022-11-16T18:53:52Z)
• Predicting Training Time Without Training [120.92623395389255]
  We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
  arXiv  Detail & Related papers  (2020-08-28T04:29:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.