Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection

• URL: http://arxiv.org/abs/2603.13070v1
• Date: Fri, 13 Mar 2026 15:16:27 GMT
• Title: Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection
• Abstract summary: Region-Aware Prompt Augmentation (RAPTA) and Attention-Driven Multimodal Copy Detection (ADMCD) are presented.<n>RAPTA uses an object detector to find salient regions and turn them into semantically grounded prompt variants.<n>ADMCD aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation.<n>Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
• Score: 53.789057575175065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State-of-the-art text-to-image diffusion models can produce impressive visuals but may memorize and reproduce training images, creating copyright and privacy risks. Existing prompt perturbations applied at inference time, such as random token insertion or embedding noise, may lower copying but often harm image-prompt alignment and overall fidelity. To address this, we introduce two complementary methods. First, Region-Aware Prompt Augmentation (RAPTA) uses an object detector to find salient regions and turn them into semantically grounded prompt variants, which are randomly sampled during training to increase diversity, while maintaining semantic alignment. Second, Attention-Driven Multimodal Copy Detection (ADMCD) aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation, and applies simple thresholded decision rules to detect copying without training with large annotated datasets. Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.

Related papers

• You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models [8.429432661292964]
  Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images.<n>We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models.<n>GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data.
  arXiv  Detail & Related papers  (2026-02-23T17:20:40Z)
• MSN: Multi-directional Similarity Network for Hand-crafted and Deep-synthesized Copy-Move Forgery Detection [41.87843079741093]
  We propose a novel two-stream model, namely Multi-directional Similarity Network (MSN), to accurate and efficient copy-move forgery detection.<n>In representation, an image is hierarchically encoded by a multi-directional CNN network, and due to the diverse augmentation in scales and rotations, the feature achieved better measures the similarity between sampled patches in two streams.<n>In localization, we design a 2-D similarity matrix based decoder, and compared with the current 1-D similarity vector based one, it makes full use of spatial information in the entire image.
  arXiv  Detail & Related papers  (2025-12-08T02:47:05Z)
• INSIGHT: An Interpretable Neural Vision-Language Framework for Reasoning of Generative Artifacts [0.0]
  Current forensic systems degrade sharply under real-world conditions.<n>Most detectors operate as opaques, offering little insight into why an image is flagged as synthetic.<n>We introduce INSIGHT, a unified framework for robust detection and transparent explanation of AI-generated images.
  arXiv  Detail & Related papers  (2025-11-27T11:43:50Z)
• Beyond Frequency: Scoring-Driven Debiasing for Object Detection via Blueprint-Prompted Image Synthesis [97.37770785712475]
  We present a generation-based debiasing framework for object detection.<n>Our method significantly narrows the performance gap for underrepresented object groups.
  arXiv  Detail & Related papers  (2025-10-21T02:19:12Z)
• FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generation [16.628211648386454]
  FICGen seeks to transfer frequency knowledge of degraded images into the latent diffusion space.<n>FICGen consistently surpasses existing L2I methods in terms of generative fidelity, alignment and downstream auxiliary trainability.
  arXiv  Detail & Related papers  (2025-09-01T04:00:22Z)
• Finding Dori: Memorization in Text-to-Image Diffusion Models Is Not Local [55.33447817350623]
  Recent mitigation efforts have focused on identifying and pruning weights responsible for triggering verbatim training data replication.<n>We challenge this assumption and demonstrate that, even after such pruning, small perturbations to the text embeddings of previously mitigated prompts can re-trigger data replication.<n>Our findings provide new insights into the nature of memorization in text-to-image DMs and inform the development of more reliable mitigations against DM memorization.
  arXiv  Detail & Related papers  (2025-07-22T15:02:38Z)
• CopyJudge: Automated Copyright Infringement Identification and Mitigation in Text-to-Image Diffusion Models [58.58208005178676]
  We propose CopyJudge, a novel automated infringement identification framework.<n>We employ an abstraction-filtration-comparison test framework to assess the likelihood of infringement.<n>We introduce a general LVLM-based mitigation strategy that automatically optimize infringing prompts.
  arXiv  Detail & Related papers  (2025-02-21T08:09:07Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive [21.49096276631859]
  Current L2I models either suffer from poor editability via text or weak alignment between the generated image and the input layout. We propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM) Specifically, we employ a segmentation-based discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout.
  arXiv  Detail & Related papers  (2024-01-16T20:31:46Z)
• Grounded Text-to-Image Synthesis with Attention Refocusing [16.9170825951175]
  We reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. We show that our proposed attention refocusing effectively improves the controllability of existing approaches.
  arXiv  Detail & Related papers  (2023-06-08T17:59:59Z)
• Revisiting Consistency Regularization for Semi-supervised Change Detection in Remote Sensing Images [60.89777029184023]
  We propose a semi-supervised CD model in which we formulate an unsupervised CD loss in addition to the supervised Cross-Entropy (CE) loss. Experiments conducted on two publicly available CD datasets show that the proposed semi-supervised CD method can reach closer to the performance of supervised CD.
  arXiv  Detail & Related papers  (2022-04-18T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.