Context-guided Responsible Data Augmentation with Diffusion Models

• URL: http://arxiv.org/abs/2503.10687v1
• Date: Wed, 12 Mar 2025 00:12:27 GMT
• Title: Context-guided Responsible Data Augmentation with Diffusion Models
• Authors: Khawar Islam, Naveed Akhtar,
• Abstract summary: We propose a text-to-image (T2I) data augmentation method, named DiffCoRe-Mix, that computes a set of generative counterparts for a training sample.<n>To preserve key semantic axes, we also filter out undesired generative samples in our augmentation process.<n>We extensively evaluate our technique on ImageNet-1K,Tiny ImageNet-200, CIFAR-100, Flowers102, CUB-Birds, Stanford Cars, and Caltech datasets.
• Score: 29.41191005466334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative diffusion models offer a natural choice for data augmentation when training complex vision models. However, ensuring reliability of their generative content as augmentation samples remains an open challenge. Despite a number of techniques utilizing generative images to strengthen model training, it remains unclear how to utilize the combination of natural and generative images as a rich supervisory signal for effective model induction. In this regard, we propose a text-to-image (T2I) data augmentation method, named DiffCoRe-Mix, that computes a set of generative counterparts for a training sample with an explicitly constrained diffusion model that leverages sample-based context and negative prompting for a reliable augmentation sample generation. To preserve key semantic axes, we also filter out undesired generative samples in our augmentation process. To that end, we propose a hard-cosine filtration in the embedding space of CLIP. Our approach systematically mixes the natural and generative images at pixel and patch levels. We extensively evaluate our technique on ImageNet-1K,Tiny ImageNet-200, CIFAR-100, Flowers102, CUB-Birds, Stanford Cars, and Caltech datasets, demonstrating a notable increase in performance across the board, achieving up to $\sim 3\%$ absolute gain for top-1 accuracy over the state-of-the-art methods, while showing comparable computational overhead. Our code is publicly available at https://github.com/khawar-islam/DiffCoRe-Mix

Related papers

• Generative Dataset Distillation using Min-Max Diffusion Model [11.433714787534582]
  This paper addresses the problem of generative dataset distillation that utilizes generative models to synthesize images. We leverage the popular diffusion model as the generator to compute a surrogate dataset, boosted by a min-max loss to control the dataset's diversity and representativeness during training. We observe a critical trade-off between the number of image samples and the image quality controlled by the diffusion steps and propose Diffusion Step Reduction to achieve optimal performance.
  arXiv  Detail & Related papers  (2025-03-24T12:41:40Z)
• Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding [84.3224556294803]
  Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. We aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Our algorithm integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future.
  arXiv  Detail & Related papers  (2024-08-15T16:47:59Z)
• Gradient Inversion of Federated Diffusion Models [4.1355611383748005]
  Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. In this paper, we study the privacy risk of gradient inversion attacks. We propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data.
  arXiv  Detail & Related papers  (2024-05-30T18:00:03Z)
• DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models [18.44432223381586]
  Recently, a number of image-mixing-based augmentation techniques have been introduced to improve the generalization of deep neural networks. In these techniques, two or more randomly selected natural images are mixed together to generate an augmented image. We propose DiffuseMix, a novel data augmentation technique that leverages a diffusion model to reshape training images.
  arXiv  Detail & Related papers  (2024-04-05T05:31:02Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• The Journey, Not the Destination: How Data Guides Diffusion Models [75.19694584942623]
  Diffusion models trained on large datasets can synthesize photo-realistic images of remarkable quality and diversity. We propose a framework that: (i) provides a formal notion of data attribution in the context of diffusion models, and (ii) allows us to counterfactually validate such attributions.
  arXiv  Detail & Related papers  (2023-12-11T08:39:43Z)
• GeNIe: Generative Hard Negative Images Through Diffusion [16.619150568764262]
  Recent advances in generative AI have enabled more sophisticated augmentation techniques that produce data resembling natural images. We introduce GeNIe, a novel augmentation method which leverages a latent diffusion model conditioned on a text prompt to generate challenging augmentations. Our experiments demonstrate the effectiveness of our novel augmentation method and its superior performance over the prior art.
  arXiv  Detail & Related papers  (2023-12-05T07:34:30Z)
• Reverse Stable Diffusion: What prompt was used to generate this image? [73.10116197883303]
  We study the task of predicting the prompt embedding given an image generated by a generative diffusion model. We propose a novel learning framework comprising a joint prompt regression and multi-label vocabulary classification objective. We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion.
  arXiv  Detail & Related papers  (2023-08-02T23:39:29Z)
• Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
  Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance. We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T12:18:20Z)
• Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
  We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification. Our generative approach to classification attains strong results on a variety of benchmarks. Our results are a step toward using generative over discriminative models for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T17:59:56Z)
• On Distillation of Guided Diffusion Models [94.95228078141626]
  We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from. For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
  arXiv  Detail & Related papers  (2022-10-06T18:03:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.