SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation

• URL: http://arxiv.org/abs/2506.19360v2
• Date: Thu, 26 Jun 2025 01:32:12 GMT
• Title: SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation
• Authors: Yunsung Chung, Yunbei Zhang, Nassir Marrouche, Jihun Hamm,
• Abstract summary: This study seeks to answer questions in PPDS: Can synthetic data effectively replace real data?<n>With a systematic evaluation of diverse methods, our study provides actionable insights into the utility-privacy tradeoffs of synthetic data generation methods.
• Score: 5.2438534296318355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advances in generative models have transformed the field of synthetic image generation for privacy-preserving data synthesis (PPDS). However, the field lacks a comprehensive survey and comparison of synthetic image generation methods across diverse settings. In particular, when we generate synthetic images for the purpose of training a classifier, there is a pipeline of generation-sampling-classification which takes private training as input and outputs the final classifier of interest. In this survey, we systematically categorize existing image synthesis methods, privacy attacks, and mitigations along this generation-sampling-classification pipeline. To empirically compare diverse synthesis approaches, we provide a benchmark with representative generative methods and use model-agnostic membership inference attacks (MIAs) as a measure of privacy risk. Through this study, we seek to answer critical questions in PPDS: Can synthetic data effectively replace real data? Which release strategy balances utility and privacy? Do mitigations improve the utility-privacy tradeoff? Which generative models perform best across different scenarios? With a systematic evaluation of diverse methods, our study provides actionable insights into the utility-privacy tradeoffs of synthetic data generation methods and guides the decision on optimal data releasing strategies for real-world applications.

Related papers

• An Empirical Study of Validating Synthetic Data for Text-Based Person Retrieval [51.10419281315848]
  We conduct an empirical study to explore the potential of synthetic data for Text-Based Person Retrieval (TBPR) research.<n>We propose an inter-class image generation pipeline, in which an automatic prompt construction strategy is introduced.<n>We develop an intra-class image augmentation pipeline, in which the generative AI models are applied to further edit the images.
  arXiv  Detail & Related papers  (2025-03-28T06:18:15Z)
• Opinion: Revisiting synthetic data classifications from a privacy perspective [42.12937192948916]
  Synthetic data is emerging as a cost-effective solution to meet the increasing data demands of AI development.<n>Traditional classification of synthetic data types does not reflect the ever-increasing methods to generate synthetic data.<n>We make a case for an alternative approach to grouping synthetic data types that better reflect privacy perspectives.
  arXiv  Detail & Related papers  (2025-03-05T13:54:13Z)
• Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data [104.30479583607918]
  2nd FRCSyn-onGoing challenge is based on the 2nd Face Recognition Challenge in the Era of Synthetic Data (FRCSyn), originally launched at CVPR 2024.<n>We focus on exploring the use of synthetic data both individually and in combination with real data to solve current challenges in face recognition.
  arXiv  Detail & Related papers  (2024-12-02T11:12:01Z)
• Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data [51.41288763521186]
  Retrieval-augmented generation (RAG) enhances the outputs of language models by integrating relevant information retrieved from external knowledge sources.<n>RAG systems may face severe privacy risks when retrieving private data.<n>We propose using synthetic data as a privacy-preserving alternative for the retrieval data.
  arXiv  Detail & Related papers  (2024-06-20T22:53:09Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• SoK: Privacy-Preserving Data Synthesis [72.92263073534899]
  This paper focuses on privacy-preserving data synthesis (PPDS) by providing a comprehensive overview, analysis, and discussion of the field. We put forth a master recipe that unifies two prominent strands of research in PPDS: statistical methods and deep learning (DL)-based methods.
  arXiv  Detail & Related papers  (2023-07-05T08:29:31Z)
• Haven't I Seen You Before? Assessing Identity Leakage in Synthetic Irises [4.142375560633827]
  This paper presents analysis for three different iris matchers at varying points in the GAN training process to diagnose where and when authentic training samples are in jeopardy of leaking through the generative process. Our results show that while most synthetic samples do not show signs of identity leakage, a handful of generated samples match authentic (training) samples nearly perfectly, with consensus across all matchers.
  arXiv  Detail & Related papers  (2022-11-03T00:34:47Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• Generation of Non-Deterministic Synthetic Face Datasets Guided by Identity Priors [19.095368725147367]
  We propose a non-deterministic method for generating mated face images by exploiting the well-structured latent space of StyleGAN. We create a new dataset of synthetic face images (SymFace) consisting of 77,034 samples including 25,919 synthetic IDs.
  arXiv  Detail & Related papers  (2021-12-07T11:08:47Z)
• Synthetic Data for Model Selection [2.4499092754102874]
  We show that synthetic data can be beneficial for model selection. We introduce a novel method to calibrate the synthetic error estimation to fit that of the real domain.
  arXiv  Detail & Related papers  (2021-05-03T09:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.