Exploring the Impact of Dataset Bias on Dataset Distillation

• URL: http://arxiv.org/abs/2403.16028v1
• Date: Sun, 24 Mar 2024 06:10:22 GMT
• Title: Exploring the Impact of Dataset Bias on Dataset Distillation
• Authors: Yao Lu, Jianyang Gu, Xuguang Chen, Saeed Vahidian, Qi Xuan,
• Abstract summary: We investigate the influence of dataset bias on Dataset Distillation (DD) DD is a technique to synthesize a smaller dataset that preserves essential information from the original dataset. Experiments demonstrate that biases present in the original dataset significantly impact the performance of the synthetic dataset.
• Score: 10.742404631413029
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dataset Distillation (DD) is a promising technique to synthesize a smaller dataset that preserves essential information from the original dataset. This synthetic dataset can serve as a substitute for the original large-scale one, and help alleviate the training workload. However, current DD methods typically operate under the assumption that the dataset is unbiased, overlooking potential bias issues within the dataset itself. To fill in this blank, we systematically investigate the influence of dataset bias on DD. To the best of our knowledge, this is the first exploration in the DD domain. Given that there are no suitable biased datasets for DD, we first construct two biased datasets, CMNIST-DD and CCIFAR10-DD, to establish a foundation for subsequent analysis. Then we utilize existing DD methods to generate synthetic datasets on CMNIST-DD and CCIFAR10-DD, and evaluate their performance following the standard process. Experiments demonstrate that biases present in the original dataset significantly impact the performance of the synthetic dataset in most cases, which highlights the necessity of identifying and mitigating biases in the original datasets during DD. Finally, we reformulate DD within the context of a biased dataset. Our code along with biased datasets are available at https://github.com/yaolu-zjut/Biased-DD.

Related papers

• Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning [10.116674195405126]
  We argue that a precise characterization of the underlying optimization problem must specify the inference task associated with the application of interest. Our formalization reveals novel applications of DD across different modeling environments. We present numerical results for two case studies important in contemporary settings.
  arXiv  Detail & Related papers  (2024-09-02T18:11:15Z)
• Distilling Long-tailed Datasets [13.330572317331198]
  We propose a novel long-tailed dataset distillation method, Long-tailed dataset Aware distillation (LAD) LAD reduces the distance between the student and the biased expert trajectories and prevents the tail class bias from being distilled to the synthetic dataset. This work pioneers the field of long-tailed dataset distillation (LTDD), marking the first effective effort to distill long-tailed datasets.
  arXiv  Detail & Related papers  (2024-08-24T15:36:36Z)
• Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation [57.6797306341115]
  We take an initial step towards understanding various matching-based DD methods from the perspective of sample difficulty. We then extend the neural scaling laws of data pruning to DD to theoretically explain these matching-based methods. We introduce the Sample Difficulty Correction (SDC) approach, designed to predominantly generate easier samples to achieve higher dataset quality.
  arXiv  Detail & Related papers  (2024-08-22T15:20:32Z)
• Mitigating Bias in Dataset Distillation [62.79454960378792]
  We study the impact of bias inside the original dataset on the performance of dataset distillation. We introduce a simple yet highly effective approach based on a sample reweighting scheme utilizing kernel density estimation.
  arXiv  Detail & Related papers  (2024-06-06T18:52:28Z)
• Importance-Aware Adaptive Dataset Distillation [53.79746115426363]
  Development of deep learning models is enabled by the availability of large-scale datasets. dataset distillation aims to synthesize a compact dataset that retains the essential information from the large original dataset. We propose an importance-aware adaptive dataset distillation (IADD) method that can improve distillation performance.
  arXiv  Detail & Related papers  (2024-01-29T03:29:39Z)
• Can pre-trained models assist in dataset distillation? [21.613468512330442]
  Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? We systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge.
  arXiv  Detail & Related papers  (2023-10-05T03:51:21Z)
• Evaluating the effect of data augmentation and BALD heuristics on distillation of Semantic-KITTI dataset [63.20765930558542]
  Active Learning has remained relatively unexplored for LiDAR perception tasks in autonomous driving datasets. We evaluate Bayesian active learning methods applied to the task of dataset distillation or core subset selection. We also study the effect of application of data augmentation within Bayesian AL based dataset distillation.
  arXiv  Detail & Related papers  (2023-02-21T13:56:47Z)
• Dataset Distillation: A Comprehensive Review [76.26276286545284]
  dataset distillation (DD) aims to derive a much smaller dataset containing synthetic samples, based on which the trained models yield performance comparable with those trained on the original dataset. This paper gives a comprehensive review and summary of recent advances in DD and its application.
  arXiv  Detail & Related papers  (2023-01-17T17:03:28Z)
• LiDAR dataset distillation within bayesian active learning framework: Understanding the effect of data augmentation [63.20765930558542]
  Active learning (AL) has re-gained attention recently to address reduction of annotation costs and dataset size. This paper performs a principled evaluation of AL based dataset distillation on (1/4th) of the large Semantic-KITTI dataset. We observe that data augmentation achieves full dataset accuracy using only 60% of samples from the selected dataset configuration.
  arXiv  Detail & Related papers  (2022-02-06T00:04:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.