Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting

• URL: http://arxiv.org/abs/2403.19898v2
• Date: Mon, 1 Apr 2024 01:27:14 GMT
• Title: Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting
• Abstract summary: Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising process. Despite the meaningful semantics generation, the existing arts suffer from the semantic discrepancy between masked and unmasked regions, since the semantically dense unmasked texture fails to be completely degraded while the masked regions turn to the pure noise in diffusion process,leading to the large discrepancy between them. We propose a novel structure-guided diffusion model named StrDiffusion,to reformulate the conventional texture denoising process under
• Score: 18.773724519657446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising process.Despite the meaningful semantics generation,the existing arts suffer from the semantic discrepancy between masked and unmasked regions, since the semantically dense unmasked texture fails to be completely degraded while the masked regions turn to the pure noise in diffusion process,leading to the large discrepancy between them. In this paper,we aim to answer how unmasked semantics guide texture denoising process;together with how to tackle the semantic discrepancy,to facilitate the consistent and meaningful semantics generation. To this end,we propose a novel structure-guided diffusion model named StrDiffusion,to reformulate the conventional texture denoising process under structure guidance to derive a simplified denoising objective for image inpainting,while revealing:1)the semantically sparse structure is beneficial to tackle semantic discrepancy in early stage, while dense texture generates reasonable semantics in late stage;2)the semantics from unmasked regions essentially offer the time-dependent structure guidance for the texture denoising process,benefiting from the time-dependent sparsity of the structure semantics.For the denoising process,a structure-guided neural network is trained to estimate the simplified denoising objective by exploiting the consistency of the denoised structure between masked and unmasked regions.Besides,we devise an adaptive resampling strategy as a formal criterion as whether structure is competent to guide the texture denoising process,while regulate their semantic correlations.Extensive experiments validate the merits of StrDiffusion over the state-of-the-arts.Our code is available at https://github.com/htyjers/StrDiffusion.

Related papers

• CogSENet: Blind Image Deblurring with Blur-Conditioned Semantic Routing and Explicit Frequency Fusion [3.7521959864518593]
  CogSENet is a dynamic, semantic-aligned reconstruction framework inspired by the eagle's visual system.<n>CogSENetoutperforms state-of-the-art deblurring methods in both visual quality and structural fidelity with fewer parameters.
  arXiv  Detail & Related papers  (2026-06-29T09:29:21Z)
• Dynamic Frequency Modulation for Controllable Text-driven Image Generation [26.61127727590205]
  modifying the original text prompt to achieve the expected semantic adjustments often results in unintended global structure changes.<n>This paper analyzes the impact of the frequency spectrum of noisy latent variables on the hierarchical emergence of the structure framework.<n>We propose a training-free frequency modulation method utilizing a frequency-dependent weighting function with dynamic decay.
  arXiv  Detail & Related papers  (2026-02-11T09:06:44Z)
• One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting [10.830941311992746]
  Text-guided image inpainting aims at reconstructing the masked regions as per text prompts.<n>We propose a null-text-null frequency-aware diffusion models, dubbed textbfNTN-Diff, for text-guided image inpainting.
  arXiv  Detail & Related papers  (2025-10-09T14:30:34Z)
• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• MaskDiffusion: Boosting Text-to-Image Consistency with Conditional Mask [84.84034179136458]
  A crucial factor leading to the text-image mismatch issue is the inadequate cross-modality relation learning. We propose an adaptive mask, which is conditioned on the attention maps and the prompt embeddings, to dynamically adjust the contribution of each text token to the image features. Our method, termed MaskDiffusion, is training-free and hot-pluggable for popular pre-trained diffusion models.
  arXiv  Detail & Related papers  (2023-09-08T15:53:37Z)
• Diffusion Model for Camouflaged Object Detection [2.592600158870236]
  We propose a diffusion-based framework for camouflaged object detection, termed diffCOD. The proposed method achieves favorable performance compared to the existing 11 state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-01T05:50:33Z)
• Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
  We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network. It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space. By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
  arXiv  Detail & Related papers  (2022-07-19T14:35:42Z)
• Synergy Between Semantic Segmentation and Image Denoising via Alternate Boosting [102.19116213923614]
  We propose a boosting network to perform denoising and segmentation alternately. We observe that not only denoising helps combat the drop of segmentation accuracy due to noise, but also pixel-wise semantic information boosts the capability of denoising. Experimental results show that the denoised image quality is improved substantially and the segmentation accuracy is improved to close to that of clean images.
  arXiv  Detail & Related papers  (2021-02-24T06:48:45Z)
• Image Inpainting Guided by Coherence Priors of Semantics and Textures [62.92586889409379]
  We introduce coherence priors between the semantics and textures which make it possible to concentrate on completing separate textures in a semantic-wise manner. We also propose two coherence losses to constrain the consistency between the semantics and the inpainted image in terms of the overall structure and detailed textures.
  arXiv  Detail & Related papers  (2020-12-15T02:59:37Z)
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [54.836331922449666]
  We propose a Semantic Guidance and Evaluation Network (SGE-Net) to update the structural priors and the inpainted image. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated. Experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-15T17:49:20Z)
• Spatial-Adaptive Network for Single Image Denoising [14.643663950015334]
  We propose a novel spatial-adaptive denoising network (SADNet) for efficient single image blind noise removal. Our method can surpass the state-of-the-art denoising methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2020-01-28T12:24:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.