Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation

• URL: http://arxiv.org/abs/2402.08200v1
• Date: Tue, 13 Feb 2024 03:55:09 GMT
• Title: Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation
• Authors: AprilPyone MaungMaung, Huy H. Nguyen, Hitoshi Kiya, Isao Echizen
• Abstract summary: We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers.
• Score: 14.016699770436961
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet.

Related papers

• Are Images Indistinguishable to Humans Also Indistinguishable to Classifiers? [39.31679737754048]
  We show that, from the perspective of neural network-based classifiers, even advanced diffusion models are still far from this goal. Our methodology naturally serves as a diagnostic tool for diffusion models by analyzing specific features of generated data. It sheds light on the model autophagy disorder and offers insights into the use of generated data.
  arXiv  Detail & Related papers  (2024-05-28T10:25:06Z)
• ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object [78.58860252442045]
  We introduce generative model as a data source for hard images that benchmark deep models' robustness. We are able to generate images with more diversified backgrounds, textures, and materials than any prior work, where we term this benchmark as ImageNet-D. Our work suggests that diffusion models can be an effective source to test vision models.
  arXiv  Detail & Related papers  (2024-03-27T17:23:39Z)
• Semantic Generative Augmentations for Few-Shot Counting [0.0]
  We investigate how synthetic data can benefit few-shot class-agnostic counting. We propose to rely on a double conditioning of Stable Diffusion with both a prompt and a density map. Our experiments show that our diversified generation strategy significantly improves the counting accuracy of two recent and performing few-shot counting models.
  arXiv  Detail & Related papers  (2023-10-26T11:42:48Z)
• Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence [88.00004819064672]
  Diffusion Hyperfeatures is a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors. Our method achieves superior performance on the SPair-71k real image benchmark.
  arXiv  Detail & Related papers  (2023-05-23T17:58:05Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
  We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification. Our generative approach to classification attains strong results on a variety of benchmarks. Our results are a step toward using generative over discriminative models for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T17:59:56Z)
• DIRE for Diffusion-Generated Image Detection [128.95822613047298]
  We propose a novel representation called DIffusion Reconstruction Error (DIRE) DIRE measures the error between an input image and its reconstruction counterpart by a pre-trained diffusion model. It provides a hint that DIRE can serve as a bridge to distinguish generated and real images.
  arXiv  Detail & Related papers  (2023-03-16T13:15:03Z)
• Detecting Images Generated by Diffusers [12.986394431694206]
  We consider images generated from captions in the MSCOCO and Wikimedia datasets using two state-of-the-art models: Stable Diffusion and GLIDE. Our experiments show that it is possible to detect the generated images using simple Multi-Layer Perceptrons. We find that incorporating the associated textual information with the images rarely leads to significant improvement in detection results.
  arXiv  Detail & Related papers  (2023-03-09T14:14:29Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet [36.48282338829549]
  In this paper, we develop a framework that allows us to systematically identify spurious features in large datasets like ImageNet. We validate our results by showing that presence of the harmful spurious feature of a class alone is sufficient to trigger the prediction of that class. We introduce SpuFix as a simple mitigation method to reduce the dependence of any ImageNet classifier on previously identified harmful spurious features.
  arXiv  Detail & Related papers  (2022-12-09T14:23:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.