Towards Adversarially Robust Dataset Distillation by Curvature Regularization

• URL: http://arxiv.org/abs/2403.10045v1
• Date: Fri, 15 Mar 2024 06:31:03 GMT
• Title: Towards Adversarially Robust Dataset Distillation by Curvature Regularization
• Authors: Eric Xue, Yijiang Li, Haoyang Liu, Yifan Shen, Haohan Wang,
• Abstract summary: We study how to embed adversarial robustness in distilled datasets, so that models trained on these datasets maintain the high accuracy and acquire better adversarial robustness. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training.
• Score: 11.463315774971857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset distillation (DD) allows datasets to be distilled to fractions of their original size while preserving the rich distributional information so that models trained on the distilled datasets can achieve a comparable accuracy while saving significant computational loads. Recent research in this area has been focusing on improving the accuracy of models trained on distilled datasets. In this paper, we aim to explore a new perspective of DD. We study how to embed adversarial robustness in distilled datasets, so that models trained on these datasets maintain the high accuracy and meanwhile acquire better adversarial robustness. We propose a new method that achieves this goal by incorporating curvature regularization into the distillation process with much less computational overhead than standard adversarial training. Extensive empirical experiments suggest that our method not only outperforms standard adversarial training on both accuracy and robustness with less computation overhead but is also capable of generating robust distilled datasets that can withstand various adversarial attacks.

Related papers

• Exploring the potential of prototype-based soft-labels data distillation for imbalanced data classification [0.0]
  Main goal is to push further the performance of prototype-based soft-labels distillation in terms of classification accuracy. Experimental studies trace the capability of the method to distill the data, but also the opportunity to act as an augmentation method.
  arXiv  Detail & Related papers  (2024-03-25T19:15:19Z)
• DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation [25.754877176280708]
  We introduce a comprehensive benchmark that is the most extensive to date for evaluating the adversarial robustness of distilled datasets in a unified way. Our benchmark significantly expands upon prior efforts by incorporating the latest advancements such as TESLA and SRe2L. We also discovered that incorporating distilled data into the training batches of the original dataset can yield to improvement of robustness.
  arXiv  Detail & Related papers  (2024-03-20T06:00:53Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness [8.432686179800543]
  We conduct extensive experiments to evaluate current state-of-the-art dataset distillation methods. We successfully use membership inference attacks to show that privacy risks still remain. This work offers a large-scale benchmarking framework for dataset distillation evaluation.
  arXiv  Detail & Related papers  (2023-05-05T08:19:27Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Dataset Distillation by Matching Training Trajectories [75.9031209877651]
  We propose a new formulation that optimize our distilled data to guide networks to a similar state as those trained on real data. Given a network, we train it for several iterations on our distilled data and optimize the distilled data with respect to the distance between the synthetically trained parameters and the parameters trained on real data. Our method handily outperforms existing methods and also allows us to distill higher-resolution visual data.
  arXiv  Detail & Related papers  (2022-03-22T17:58:59Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Churn Reduction via Distillation [54.5952282395487]
  We show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines.
  arXiv  Detail & Related papers  (2021-06-04T18:03:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.