Latent Feature-Guided Diffusion Models for Shadow Removal

• URL: http://arxiv.org/abs/2312.02156v1
• Date: Mon, 4 Dec 2023 18:59:55 GMT
• Title: Latent Feature-Guided Diffusion Models for Shadow Removal
• Authors: Kangfu Mei and Luis Figueroa and Zhe Lin and Zhihong Ding and Scott Cohen and Vishal M. Patel
• Abstract summary: We propose the use of diffusion models as they offer a promising approach to gradually refine the details of shadow regions during the diffusion process. Our method improves this process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images. We demonstrate the effectiveness of our approach which outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset.
• Score: 50.02857194218859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering textures under shadows has remained a challenging problem due to the difficulty of inferring shadow-free scenes from shadow images. In this paper, we propose the use of diffusion models as they offer a promising approach to gradually refine the details of shadow regions during the diffusion process. Our method improves this process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images, thus avoiding the limitation of conventional methods that condition on degraded images only. Additionally, we propose to alleviate potential local optima during training by fusing noise features with the diffusion network. We demonstrate the effectiveness of our approach which outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset. Further, we explore instance-level shadow removal, where our model outperforms the previous best method by 82% in terms of RMSE on the DESOBA dataset.

Related papers

• Detail-Preserving Latent Diffusion for Stable Shadow Removal [24.18957090960958]
  We propose a two-stage fine-tuning pipeline to adapt the Stable Diffusion model for stable and efficient shadow removal. Experimental results show that our method outperforms state-of-the-art shadow removal techniques.
  arXiv  Detail & Related papers  (2024-12-23T15:06:46Z)
• Controlling the Latent Diffusion Model for Generative Image Shadow Removal via Residual Generation [18.514154007772337]
  Large-scale generative models often generate diverse, realistic details without adequate focus on fidelity. This paper utilizes diffusion models to generate and refine image residuals. To revent the accumulation of errors during the generation process, a crosstimestep self-enhancement training strategy is proposed.
  arXiv  Detail & Related papers  (2024-12-03T09:38:14Z)
• Generative Portrait Shadow Removal [27.98144439007323]
  We introduce a high-fidelity portrait shadow removal model that can effectively enhance the image of a portrait. Our method also demonstrates robustness to diverse subjects captured in real environments.
  arXiv  Detail & Related papers  (2024-10-07T22:09:22Z)
• Diff-Shadow: Global-guided Diffusion Model for Shadow Removal [46.41983327564438]
  We propose Diff-Shadow, a global-guided diffusion model for shadow removal. We combine the advantages of diffusion models and global guidance to achieve shadow-free restoration. Our method achieves a significant improvement in terms of PSNR, increasing from 32.33dB to 33.69dB on the ISTD dataset.
  arXiv  Detail & Related papers  (2024-07-23T06:42:55Z)
• ReNoise: Real Image Inversion Through Iterative Noising [62.96073631599749]
  We introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models.
  arXiv  Detail & Related papers  (2024-03-21T17:52:08Z)
• Efficient Diffusion Model for Image Restoration by Residual Shifting [63.02725947015132]
  This study proposes a novel and efficient diffusion model for image restoration. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Our method achieves superior or comparable performance to current state-of-the-art methods on three classical IR tasks.
  arXiv  Detail & Related papers  (2024-03-12T05:06:07Z)
• ExposureDiffusion: Learning to Expose for Low-light Image Enhancement [87.08496758469835]
  This work addresses the issue by seamlessly integrating a diffusion model with a physics-based exposure model. Our method obtains significantly improved performance and reduced inference time compared with vanilla diffusion models. The proposed framework can work with both real-paired datasets, SOTA noise models, and different backbone networks.
  arXiv  Detail & Related papers  (2023-07-15T04:48:35Z)
• Low-Light Image Enhancement with Wavelet-based Diffusion Models [50.632343822790006]
  Diffusion models have achieved promising results in image restoration tasks, yet suffer from time-consuming, excessive computational resource consumption, and unstable restoration. We propose a robust and efficient Diffusion-based Low-Light image enhancement approach, dubbed DiffLL.
  arXiv  Detail & Related papers  (2023-06-01T03:08:28Z)
• ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal [74.86415440438051]
  We propose a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. Our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.
  arXiv  Detail & Related papers  (2022-12-09T07:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.