Real-Fake: Effective Training Data Synthesis Through Distribution Matching

• URL: http://arxiv.org/abs/2310.10402v2
• Date: Wed, 20 Mar 2024 12:52:10 GMT
• Title: Real-Fake: Effective Training Data Synthesis Through Distribution Matching
• Authors: Jianhao Yuan, Jie Zhang, Shuyang Sun, Philip Torr, Bo Zhao,
• Abstract summary: We analyze the principles underlying training data synthesis for supervised learning. We demonstrate the effectiveness of our synthetic data across diverse image classification tasks. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent to 1 X the original real data size.
• Score: 16.499008884926337
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic training data has gained prominence in numerous learning tasks and scenarios, offering advantages such as dataset augmentation, generalization evaluation, and privacy preservation. Despite these benefits, the efficiency of synthetic data generated by current methodologies remains inferior when training advanced deep models exclusively, limiting its practical utility. To address this challenge, we analyze the principles underlying training data synthesis for supervised learning and elucidate a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy. Through extensive experiments, we demonstrate the effectiveness of our synthetic data across diverse image classification tasks, both as a replacement for and augmentation to real datasets, while also benefits such as out-of-distribution generalization, privacy preservation, and scalability. Specifically, we achieve 70.9% top1 classification accuracy on ImageNet1K when training solely with synthetic data equivalent to 1 X the original real data size, which increases to 76.0% when scaling up to 10 X synthetic data.

Related papers

• Provably Improving Generalization of Few-Shot Models with Synthetic Data [15.33628135372502]
  We develop a theoretical framework that quantifies the impact of distribution discrepancies on supervised learning.<n>We propose a novel theoretical-based algorithm that integrates prototype learning to optimize both data partitioning and model training.
  arXiv  Detail & Related papers  (2025-05-30T03:59:45Z)
• Scaling Laws of Synthetic Data for Language Models [132.67350443447611]
  We introduce SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets. Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm.
  arXiv  Detail & Related papers  (2025-03-25T11:07:12Z)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops [55.07063067759609]
  High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs) Some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding.
  arXiv  Detail & Related papers  (2025-02-26T06:18:13Z)
• SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data [44.304022773272415]
  We introduce SynFER, a novel framework for synthesizing facial expression image data based on high-level textual descriptions. We propose a semantic guidance technique to steer the generation process and a pseudo-label generator to help rectify the facial expression labels. Our approach achieves a 67.23% classification accuracy on AffectNet when training solely with synthetic data equivalent to the AffectNet training set size.
  arXiv  Detail & Related papers  (2024-10-13T14:58:21Z)
• Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models [89.88010750772413]
  Synthetic data has been proposed as a solution to address the issue of high-quality data scarcity in the training of large language models (LLMs) Our work delves into these specific flaws associated with question-answer (Q-A) pairs, a prevalent type of synthetic data, and presents a method based on unlearning techniques to mitigate these flaws. Our work has yielded key insights into the effective use of synthetic data, aiming to promote more robust and efficient LLM training.
  arXiv  Detail & Related papers  (2024-06-18T08:38:59Z)
• On the Equivalency, Substitutability, and Flexibility of Synthetic Data [9.459709213597707]
  We investigate the equivalency of synthetic data to real-world data, the substitutability of synthetic data for real data, and the flexibility of synthetic data generators. Our results suggest that synthetic data not only enhances model performance but also demonstrates substitutability for real data, with 60% to 80% replacement without performance loss.
  arXiv  Detail & Related papers  (2024-03-24T17:21:32Z)
• Learning from Synthetic Data for Visual Grounding [55.21937116752679]
  We show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models. Data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively.
  arXiv  Detail & Related papers  (2024-03-20T17:59:43Z)
• Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmark [56.8042116967334]
  Synthetic data serves as an alternative in training machine learning models. ensuring that synthetic data mirrors the complex nuances of real-world data is a challenging task. This paper explores the potential of integrating data-centric AI techniques to guide the synthetic data generation process.
  arXiv  Detail & Related papers  (2023-10-25T20:32:02Z)
• Bridging the Gap: Enhancing the Utility of Synthetic Data via Post-Processing Techniques [7.967995669387532]
  generative models have emerged as a promising solution for generating synthetic datasets that can replace or augment real-world data. We propose three novel post-processing techniques to improve the quality and diversity of the synthetic dataset. Experiments show that Gap Filler (GaFi) effectively reduces the gap with real-accuracy scores to an error of 2.03%, 1.78%, and 3.99% on the Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets, respectively.
  arXiv  Detail & Related papers  (2023-05-17T10:50:38Z)
• A Study on Improving Realism of Synthetic Data for Machine Learning [6.806559012493756]
  This work aims to train and evaluate a synthetic-to-real generative model that transforms the synthetic renderings into more realistic styles on general-purpose datasets conditioned with unlabeled real-world data.
  arXiv  Detail & Related papers  (2023-04-24T21:41:54Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation [54.899751055620904]
  We study the phenomenon of memorization with turn-over dropout, an efficient method to estimate influence and memorization, for data with true labels (real data) and data with random labels (random data) Our main findings are: (i) For both real data and random data, the optimization of easy examples (e.g., real data) and difficult examples (e.g., random data) are conducted by the network simultaneously, with easy ones at a higher speed; (ii) For real data, a correct difficult example in the training dataset is more informative than an easy one.
  arXiv  Detail & Related papers  (2021-12-16T11:34:23Z)
• Up to 100x Faster Data-free Knowledge Distillation [52.666615987503995]
  We introduce FastDFKD, which allows us to accelerate DFKD by a factor of orders of magnitude. Unlike prior methods that optimize a set of data independently, we propose to learn a meta-synthesizer that seeks common features. FastDFKD achieves data synthesis within only a few steps, significantly enhancing the efficiency of data-free training.
  arXiv  Detail & Related papers  (2021-12-12T14:56:58Z)
• A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
  Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks. Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models. We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
  arXiv  Detail & Related papers  (2021-08-25T02:29:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.