One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting

• URL: http://arxiv.org/abs/2510.08273v5
• Date: Mon, 27 Oct 2025 00:08:58 GMT
• Title: One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting
• Abstract summary: Text-guided image inpainting aims at reconstructing the masked regions as per text prompts.<n>We propose a null-text-null frequency-aware diffusion models, dubbed textbfNTN-Diff, for text-guided image inpainting.
• Score: 10.830941311992746
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either of them to be remedied. Such facts, as we observed, stem from the entanglement of the hybrid (e.g., mid-and-low) frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denoising process. In this paper, we propose a null-text-null frequency-aware diffusion models, dubbed \textbf{NTN-Diff}, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent two challenges in a row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequency band is progressively denoised to be semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text-guided denoising process at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from https://github.com/htyjers/NTN-Diff.

Related papers

• AsyncPatch Diffusion: spatially-flexible image generation [24.854661299695156]
  AsyncPatch Diffusion is a joint-diffusion framework that assigns distinct noise levels to different input dimensions.<n>We show how this asynchronous corruption defines a valid generative process.<n>We demonstrate adaptive generation strategies including uncertainty-guided acceleration and autoregressive sampling.
  arXiv  Detail & Related papers  (2026-06-05T09:16:51Z)
• Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection [53.789057575175065]
  Region-Aware Prompt Augmentation (RAPTA) and Attention-Driven Multimodal Copy Detection (ADMCD) are presented.<n>RAPTA uses an object detector to find salient regions and turn them into semantically grounded prompt variants.<n>ADMCD aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation.<n>Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
  arXiv  Detail & Related papers  (2026-03-13T15:16:27Z)
• InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Models [27.206678799411645]
  InfSplign is a training-free inference-time method for text-to-image models.<n>It improves spatial alignment by adjusting the noise through a compound loss in every denoising step.<n>It achieves substantial performance gains over the strongest existing inference-time baselines.
  arXiv  Detail & Related papers  (2025-12-19T17:52:43Z)
• Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation [48.203403471536866]
  We propose a novel framework that allocates distinct timesteps to different pixels and reformulates the pixel-wise denoising process.<n>Our experiments demonstrate that our asynchronous diffusion models can significantly improve text-to-image alignment across diverse prompts.
  arXiv  Detail & Related papers  (2025-10-06T05:45:56Z)
• Blind-Spot Guided Diffusion for Self-supervised Real-World Denoising [55.099717395320276]
  Blind-Spot Guided Diffusion is a novel self-supervised framework for real-world image denoising.<n>Our approach addresses two major challenges: the limitations of blind-spot networks (BSNs) and the difficulty of adapting diffusion models to self-supervised denoising.
  arXiv  Detail & Related papers  (2025-09-19T15:35:07Z)
• Be Decisive: Noise-Induced Layouts for Multi-Subject Generation [56.80513553424086]
  Complex prompts lead to subject leakage, causing inaccuracies in quantities, attributes, and visual features.<n>We introduce a new approach that predicts a spatial layout aligned with the prompt, derived from the initial noise, and refines it throughout the denoising process.<n>Our method employs a small neural network to predict and refine the evolving noise-induced layout at each denoising step.
  arXiv  Detail & Related papers  (2025-05-27T17:54:24Z)
• PiCo: Enhancing Text-Image Alignment with Improved Noise Selection and Precise Mask Control in Diffusion Models [10.767325147254574]
  We propose PiCo (Pick-and-Control), a novel training-free approach with two key components to tackle these two factors.<n>First, we develop a noise selection module to assess the quality of the random noise and determine whether the noise is suitable for the target text.<n>Second, we introduce a referring mask module to generate pixel-level masks and to precisely modulate the cross-attention maps.
  arXiv  Detail & Related papers  (2025-05-06T05:38:13Z)
• Towards Understanding the Working Mechanism of Text-to-Image Diffusion Model [57.24046436423511]
  Recently, the strong latent Diffusion Probabilistic Model (DPM) has been applied to high-quality Text-to-Image (T2I) generation. We explore the mechanism behind DPM by examining the intermediate statuses during the gradual denoising generation process. We propose to apply this observation to accelerate the process of T2I generation by properly removing text guidance.
  arXiv  Detail & Related papers  (2024-05-24T08:12:41Z)
• Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting [18.773724519657446]
  Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising process. Despite the meaningful semantics generation, the existing arts suffer from the semantic discrepancy between masked and unmasked regions, since the semantically dense unmasked texture fails to be completely degraded while the masked regions turn to the pure noise in diffusion process,leading to the large discrepancy between them. We propose a novel structure-guided diffusion model named StrDiffusion,to reformulate the conventional texture denoising process under
  arXiv  Detail & Related papers  (2024-03-29T00:40:12Z)
• Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation [11.80682025950519]
  In this work, we investigate the diffusion (physics) in diffusion (machine learning) properties. We propose our Cyclic One-Way Diffusion (COW) method to control the direction of diffusion phenomenon. Our method provides a novel perspective to understand the task needs and is applicable to a wider range of customization scenarios.
  arXiv  Detail & Related papers  (2023-06-14T05:25:06Z)
• Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models [43.83732051916894]
  We propose COPAINT, which can coherently inpaint the whole image without introducing mismatches. COPAINT also uses the Bayesian framework to jointly modify both revealed and unrevealed regions. Our experiments verify that COPAINT can outperform the existing diffusion-based methods under both objective and subjective metrics.
  arXiv  Detail & Related papers  (2023-04-06T18:35:13Z)
• Region-Aware Diffusion for Zero-shot Text-driven Image Editing [78.58917623854079]
  We propose a novel region-aware diffusion model (RDM) for entity-level image editing. To strike a balance between image fidelity and inference speed, we design the intensive diffusion pipeline. The results show that RDM outperforms the previous approaches in terms of visual quality, overall harmonization, non-editing region content preservation, and text-image semantic consistency.
  arXiv  Detail & Related papers  (2023-02-23T06:20:29Z)
• FT-TDR: Frequency-guided Transformer and Top-Down Refinement Network for Blind Face Inpainting [77.78305705925376]
  Blind face inpainting refers to the task of reconstructing visual contents without explicitly indicating the corrupted regions in a face image. We propose a novel two-stage blind face inpainting method named Frequency-guided Transformer and Top-Down Refinement Network (FT-TDR) to tackle these challenges.
  arXiv  Detail & Related papers  (2021-08-10T03:12:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.