Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2510.08625v1
• Date: Wed, 08 Oct 2025 10:37:29 GMT
• Title: Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
• Authors: Hyeonggeun Han, Sehwan Kim, Hyungjun Joo, Sangwoo Hong, Jungwoo Lee,
• Abstract summary: We show that the initial noise sample plays a crucial role in determining when this escape occurs.<n>We propose two mitigation strategies that adjust the initial noise-either collectively or individually-to find and utilize initial samples that encourage earlier basin escape.
• Score: 10.935602641612888
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite their impressive generative capabilities, text-to-image diffusion models often memorize and replicate training data, prompting serious concerns over privacy and copyright. Recent work has attributed this memorization to an attraction basin-a region where applying classifier-free guidance (CFG) steers the denoising trajectory toward memorized outputs-and has proposed deferring CFG application until the denoising trajectory escapes this basin. However, such delays often result in non-memorized images that are poorly aligned with the input prompts, highlighting the need to promote earlier escape so that CFG can be applied sooner in the denoising process. In this work, we show that the initial noise sample plays a crucial role in determining when this escape occurs. We empirically observe that different initial samples lead to varying escape times. Building on this insight, we propose two mitigation strategies that adjust the initial noise-either collectively or individually-to find and utilize initial samples that encourage earlier basin escape. These approaches significantly reduce memorization while preserving image-text alignment.

Related papers

• You Don't Need All That Attention: Surgical Memorization Mitigation in Text-to-Image Diffusion Models [8.429432661292964]
  Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images.<n>We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models.<n>GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data.
  arXiv  Detail & Related papers  (2026-02-23T17:20:40Z)
• Noise as a Probe: Membership Inference Attacks on Diffusion Models Leveraging Initial Noise [51.179816451161635]
  Diffusion models have achieved remarkable progress in image generation, but their increasing deployment raises serious concerns about privacy.<n>In this work, we utilize a critical yet overlooked vulnerability: the widely used noise schedules fail to fully eliminate semantic information in the images.<n>We propose a simple yet effective membership inference attack, which injects semantic information into the initial noise and infers membership by analyzing the model's generation result.
  arXiv  Detail & Related papers  (2026-01-29T12:29:01Z)
• Memorization Control in Diffusion Models from Denoising-centric Perspective [0.6741942263052466]
  Controlling memorization in diffusion models is critical for applications that require generated data to closely match the training distribution.<n>We show that uniform timestep sampling leads to unequal learning contributions across denoising steps due to differences in signal to noise ratio.<n>We propose a timestep sampling strategy that explicitly controls where learning occurs along the denoising trajectory.
  arXiv  Detail & Related papers  (2026-01-29T07:16:54Z)
• CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Models [60.610268549138375]
  Diffusion models can unintentionally reproduce training examples, raising privacy and copyright concerns.<n>We introduce CAPTAIN, a training-free framework that mitigates memorization by directly modifying latent features during denoising.
  arXiv  Detail & Related papers  (2025-12-11T14:01:47Z)
• Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models [9.683618735282414]
  In text-to-image generation, different initial noises induce distinct denoising paths with a pretrained Stable Diffusion (SD) model.<n>While this pattern could output diverse images, some of them may fail to align well with the prompt.<n>We propose a noise projector that applies text-conditioned refinement to the initial noise before denoising.
  arXiv  Detail & Related papers  (2025-10-16T10:14:34Z)
• How Diffusion Models Memorize [26.711679643772623]
  diffusion models can memorize training data, raising serious privacy and copyright concerns.<n>We show memorization is driven by the overestimation of training samples during early denoising.
  arXiv  Detail & Related papers  (2025-09-30T03:03:27Z)
• Exploiting the Exact Denoising Posterior Score in Training-Free Guidance of Diffusion Models [0.0]
  A popular class of methods, based on Diffusion Posterior Sampling (DPS), attempts to approximate the intractable posterior score function directly.<n>We present a novel expression for the exact posterior score for purely denoising tasks that is tractable in terms of the unconditional score function.<n>We demonstrate that these step sizes are transferable to related inverse problems such as colorization, random inpainting, and super resolution.
  arXiv  Detail & Related papers  (2025-06-16T15:43:28Z)
• Active Adversarial Noise Suppression for Image Forgery Localization [56.98050814363447]
  We introduce an Adversarial Noise Suppression Module (ANSM) that generate a defensive perturbation to suppress the attack effect of adversarial noise.<n>To our best knowledge, this is the first report of adversarial defense in image forgery localization tasks.
  arXiv  Detail & Related papers  (2025-06-15T14:53:27Z)
• Be Decisive: Noise-Induced Layouts for Multi-Subject Generation [56.80513553424086]
  Complex prompts lead to subject leakage, causing inaccuracies in quantities, attributes, and visual features.<n>We introduce a new approach that predicts a spatial layout aligned with the prompt, derived from the initial noise, and refines it throughout the denoising process.<n>Our method employs a small neural network to predict and refine the evolving noise-induced layout at each denoising step.
  arXiv  Detail & Related papers  (2025-05-27T17:54:24Z)
• Rethinking and Defending Protective Perturbation in Personalized Diffusion Models [21.30373461975769]
  We study the fine-tuning process of personalized diffusion models (PDMs) through the lens of shortcut learning. PDMs are susceptible to minor adversarial perturbations, leading to significant degradation when fine-tuned on corrupted datasets. We propose a systematic defense framework that includes data purification and contrastive decoupling learning.
  arXiv  Detail & Related papers  (2024-06-27T07:14:14Z)
• InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization [27.508861002013358]
  InitNO is a paradigm that refines the initial noise in semantically-faithful images. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts.
  arXiv  Detail & Related papers  (2024-04-06T14:56:59Z)
• Representing Noisy Image Without Denoising [91.73819173191076]
  Fractional-order Moments in Radon space (FMR) is designed to derive robust representation directly from noisy images. Unlike earlier integer-order methods, our work is a more generic design taking such classical methods as special cases.
  arXiv  Detail & Related papers  (2023-01-18T10:13:29Z)
• Salvage Reusable Samples from Noisy Data for Robust Learning [70.48919625304]
  We propose a reusable sample selection and correction approach, termed as CRSSC, for coping with label noise in training deep FG models with web images. Our key idea is to additionally identify and correct reusable samples, and then leverage them together with clean examples to update the networks.
  arXiv  Detail & Related papers  (2020-08-06T02:07:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.