Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement

• URL: http://arxiv.org/abs/2309.07254v4
• Date: Tue, 23 Jan 2024 20:43:50 GMT
• Title: Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement
• Authors: Chenghao Li, Dake Chen, Yuke Zhang, Peter A. Beerel
• Abstract summary: We introduce a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. We leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models.
• Score: 7.9911486976035215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to `replicate' training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5% compared to the original diffusion model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.

Related papers

• Gradient Inversion of Federated Diffusion Models [4.1355611383748005]
  Diffusion models are becoming defector generative models, which generate exceptionally high-resolution image data. In this paper, we study the privacy risk of gradient inversion attacks. We propose a triple-optimization GIDM+ that coordinates the optimization of the unknown data.
  arXiv  Detail & Related papers  (2024-05-30T18:00:03Z)
• Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy [36.156856772794065]
  We propose a conditional overfitting phenomenon in text-to-image diffusion models. Our method significantly outperforms previous methods across various data and dataset scales.
  arXiv  Detail & Related papers  (2024-05-23T17:09:51Z)
• Text Diffusion with Reinforced Conditioning [92.17397504834825]
  This paper thoroughly analyzes text diffusion models and uncovers two significant limitations: degradation of self-conditioning during training and misalignment between training and sampling. Motivated by our findings, we propose a novel Text Diffusion model called TREC, which mitigates the degradation with Reinforced Conditioning and the misalignment by Time-Aware Variance Scaling.
  arXiv  Detail & Related papers  (2024-02-19T09:24:02Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Memory Triggers: Unveiling Memorization in Text-To-Image Generative Models through Word-Level Duplication [16.447035745151428]
  Diffusion-based models have revolutionized text-to-image synthesis with their ability to produce high-quality, high-resolution images. These models also raise concerns due to their tendency to replicate exact training samples, posing privacy risks and enabling adversarial attacks. This paper focuses on two distinct and underexplored types of duplication that lead to replication during inference in diffusion-based models.
  arXiv  Detail & Related papers  (2023-12-06T18:54:44Z)
• Prompt-Based Exemplar Super-Compression and Regeneration for Class-Incremental Learning [21.136513495039242]
  We introduce PESCR, a novel approach that substantially increases the quantity and enhances the diversity of exemplars. Images are compressed into visual and textual prompts, which are saved instead of the original images. In subsequent phases, diverse exemplars are regenerated by the diffusion model.
  arXiv  Detail & Related papers  (2023-11-30T05:59:31Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion. This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement. We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Diffusion Models for Image Restoration and Enhancement -- A Comprehensive Survey [96.99328714941657]
  We present a comprehensive review of recent diffusion model-based methods on image restoration. We classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR. We propose five potential and challenging directions for the future research of diffusion model-based IR.
  arXiv  Detail & Related papers  (2023-08-18T08:40:38Z)
• LLDiffusion: Learning Degradation Representations in Diffusion Models for Low-Light Image Enhancement [118.83316133601319]
  Current deep learning methods for low-light image enhancement (LLIE) typically rely on pixel-wise mapping learned from paired data. We propose a degradation-aware learning scheme for LLIE using diffusion models, which effectively integrates degradation and image priors into the diffusion process.
  arXiv  Detail & Related papers  (2023-07-27T07:22:51Z)
• Understanding and Mitigating Copying in Diffusion Models [53.03978584040557]
  Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user.
  arXiv  Detail & Related papers  (2023-05-31T17:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.