Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement

• URL: http://arxiv.org/abs/2309.07254v4
• Date: Tue, 23 Jan 2024 20:43:50 GMT
• Title: Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement
• Authors: Chenghao Li, Dake Chen, Yuke Zhang, Peter A. Beerel
• Abstract summary: We introduce a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. We leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models.
• Score: 7.9911486976035215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to `replicate' training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5% compared to the original diffusion model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.

Related papers

• Memorized Images in Diffusion Models share a Subspace that can be Located and Deleted [15.162296378581853]
  Large-scale text-to-image diffusion models excel in generating high-quality images from textual inputs. Concerns arise as research indicates their tendency to memorize and replicate training data. Efforts within the text-to-image community to address memorization explore causes such as data duplication, replicated captions, or trigger tokens.
  arXiv  Detail & Related papers  (2024-06-01T15:47:13Z)
• Gradient Inversion of Federated Diffusion Models [4.1355611383748005]
  Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. In this paper, we study the privacy risk of gradient inversion attacks. We propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data.
  arXiv  Detail & Related papers  (2024-05-30T18:00:03Z)
• Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy [36.156856772794065]
  We present a conditional overfitting phenomenon in text-to-image diffusion models, indicating that these models tend to overfit the conditional distribution of images given the text. We derive an analytical indicator, namely Conditional Likelihood Discrepancy (CLiD), to perform membership inference. Experimental results demonstrate that our method significantly outperforms previous methods across various data distributions and scales.
  arXiv  Detail & Related papers  (2024-05-23T17:09:51Z)
• Diffusion Models Trained with Large Data Are Transferable Visual Models [49.84679952948808]
  We show that it is possible to achieve remarkable transferable performance on fundamental vision perception tasks using a moderate amount of target data. Results showcase the remarkable transferability of the backbone of diffusion models across diverse tasks and real-world datasets.
  arXiv  Detail & Related papers  (2024-03-10T04:23:24Z)
• Text Diffusion with Reinforced Conditioning [92.17397504834825]
  This paper thoroughly analyzes text diffusion models and uncovers two significant limitations: degradation of self-conditioning during training and misalignment between training and sampling. Motivated by our findings, we propose a novel Text Diffusion model called TREC, which mitigates the degradation with Reinforced Conditioning and the misalignment by Time-Aware Variance Scaling.
  arXiv  Detail & Related papers  (2024-02-19T09:24:02Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Memory Triggers: Unveiling Memorization in Text-To-Image Generative Models through Word-Level Duplication [16.447035745151428]
  Diffusion-based models have revolutionized text-to-image synthesis with their ability to produce high-quality, high-resolution images. These models also raise concerns due to their tendency to replicate exact training samples, posing privacy risks and enabling adversarial attacks. This paper focuses on two distinct and underexplored types of duplication that lead to replication during inference in diffusion-based models.
  arXiv  Detail & Related papers  (2023-12-06T18:54:44Z)
• Diffusion Models for Image Restoration and Enhancement -- A Comprehensive Survey [96.99328714941657]
  We present a comprehensive review of recent diffusion model-based methods on image restoration. We classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR. We propose five potential and challenging directions for the future research of diffusion model-based IR.
  arXiv  Detail & Related papers  (2023-08-18T08:40:38Z)
• LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement [118.83316133601319]
  Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data. We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
  arXiv  Detail & Related papers  (2023-07-27T07:22:51Z)
• Understanding and Mitigating Copying in Diffusion Models [53.03978584040557]
  Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user.
  arXiv  Detail & Related papers  (2023-05-31T17:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.