An In-Depth Analysis of Data Reduction Methods for Sustainable Deep Learning

• URL: http://arxiv.org/abs/2403.15150v1
• Date: Fri, 22 Mar 2024 12:06:40 GMT
• Title: An In-Depth Analysis of Data Reduction Methods for Sustainable Deep Learning
• Authors: Víctor Toscano-Durán, Javier Perera-Lago, Eduardo Paluzo-Hidalgo, Rocío Gonzalez-Diaz, Miguel Ángel Gutierrez-Naranjo, Matteo Rucco,
• Abstract summary: We present up to eight different methods to reduce the size of a training dataset. We also develop a Python package to apply them. We experimentally compare how these data reduction methods affect the representativeness of the reduced dataset.
• Score: 0.15833270109954137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, Deep Learning has gained popularity for its ability to solve complex classification tasks, increasingly delivering better results thanks to the development of more accurate models, the availability of huge volumes of data and the improved computational capabilities of modern computers. However, these improvements in performance also bring efficiency problems, related to the storage of datasets and models, and to the waste of energy and time involved in both the training and inference processes. In this context, data reduction can help reduce energy consumption when training a deep learning model. In this paper, we present up to eight different methods to reduce the size of a tabular training dataset, and we develop a Python package to apply them. We also introduce a representativeness metric based on topology to measure how similar are the reduced datasets and the full training dataset. Additionally, we develop a methodology to apply these data reduction methods to image datasets for object detection tasks. Finally, we experimentally compare how these data reduction methods affect the representativeness of the reduced dataset, the energy consumption and the predictive performance of the model.

Related papers

• Large-Scale Dataset Pruning in Adversarial Training through Data Importance Extrapolation [1.3124513975412255]
  We propose a new data pruning strategy based on extrapolating data importance scores from a small set of data to a larger set. In an empirical evaluation, we demonstrate that extrapolation-based pruning can efficiently reduce dataset size while maintaining robustness.
  arXiv  Detail & Related papers  (2024-06-19T07:23:51Z)
• Improvement of Applicability in Student Performance Prediction Based on Transfer Learning [2.3290007848431955]
  This study proposes a method to improve prediction accuracy by employing transfer learning techniques on the dataset with varying distributions. The model was trained and evaluated to enhance its generalization ability and prediction accuracy. Experiments demonstrated that this approach excels in reducing Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) The results demonstrate that freezing more layers improves performance for complex and noisy data, whereas freezing fewer layers is more effective for simpler and larger datasets.
  arXiv  Detail & Related papers  (2024-06-01T13:09:05Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection [18.683805940232485]
  We introduce a novel method that employs core subset selection for reweighting. By focusing on a strategically selected coreset, our approach offers a robust representation. The re-calibrated weights are then mapped back to and propagated across the entire dataset.
  arXiv  Detail & Related papers  (2024-03-18T18:30:22Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Efficiently Robustify Pre-trained Models [18.392732966487582]
  robustness of large scale models towards real-world settings is still a less-explored topic. We first benchmark the performance of these models under different perturbations and datasets. We then discuss on how complete model fine-tuning based existing robustification schemes might not be a scalable option given very large scale networks.
  arXiv  Detail & Related papers  (2023-09-14T08:07:49Z)
• Improved Distribution Matching for Dataset Condensation [91.55972945798531]
  We propose a novel dataset condensation method based on distribution matching. Our simple yet effective method outperforms most previous optimization-oriented methods with much fewer computational resources.
  arXiv  Detail & Related papers  (2023-07-19T04:07:33Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Towards Robust Dataset Learning [90.2590325441068]
  We propose a principled, tri-level optimization to formulate the robust dataset learning problem. Under an abstraction model that characterizes robust vs. non-robust features, the proposed method provably learns a robust dataset.
  arXiv  Detail & Related papers  (2022-11-19T17:06:10Z)
• Automatic Data Augmentation via Invariance-Constrained Learning [94.27081585149836]
  Underlying data structures are often exploited to improve the solution of learning tasks. Data augmentation induces these symmetries during training by applying multiple transformations to the input data. This work tackles these issues by automatically adapting the data augmentation while solving the learning task.
  arXiv  Detail & Related papers  (2022-09-29T18:11:01Z)
• Dataset Condensation with Gradient Matching [36.14340188365505]
  We propose a training set synthesis technique for data-efficient learning, called dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-10T16:30:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.