Detail-Preserving Latent Diffusion for Stable Shadow Removal

• URL: http://arxiv.org/abs/2412.17630v1
• Date: Mon, 23 Dec 2024 15:06:46 GMT
• Title: Detail-Preserving Latent Diffusion for Stable Shadow Removal
• Authors: Jiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu,
• Abstract summary: We propose a two-stage fine-tuning pipeline to adapt the Stable Diffusion model for stable and efficient shadow removal. Experimental results show that our method outperforms state-of-the-art shadow removal techniques.
• Score: 24.18957090960958
• License:
• Abstract: Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods.

Related papers

• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• Semantic-guided Adversarial Diffusion Model for Self-supervised Shadow Removal [5.083330121710086]
  GAN-based training often faces issues such as mode collapse and unstable optimization. We propose a semantic-guided adversarial diffusion framework for self-supervised shadow removal. We conduct experiments on multiple public datasets, and the experimental results demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-07-01T09:14:38Z)
• DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery [71.6345505427213]
  DPMesh is an innovative framework for occluded human mesh recovery. It capitalizes on the profound diffusion prior about object structure and spatial relationships embedded in a pre-trained text-to-image diffusion model.
  arXiv  Detail & Related papers  (2024-04-01T18:59:13Z)
• Latent Feature-Guided Diffusion Models for Shadow Removal [50.02857194218859]
  We propose the use of diffusion models as they offer a promising approach to gradually refine the details of shadow regions during the diffusion process. Our method improves this process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images. We demonstrate the effectiveness of our approach which outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset.
  arXiv  Detail & Related papers  (2023-12-04T18:59:55Z)
• Progressive Recurrent Network for Shadow Removal [99.1928825224358]
  Single-image shadow removal is a significant task that is still unresolved. Most existing deep learning-based approaches attempt to remove the shadow directly, which can not deal with the shadow well. We propose a simple but effective Progressive Recurrent Network (PRNet) to remove the shadow progressively.
  arXiv  Detail & Related papers  (2023-11-01T11:42:45Z)
• Deshadow-Anything: When Segment Anything Model Meets Zero-shot shadow removal [8.555176637147648]
  We develop Deshadow-Anything, considering the generalization of large-scale datasets, to achieve image shadow removal. The diffusion model can diffuse along the edges and textures of an image, helping to remove shadows while preserving the details of the image. Experiments on shadow removal tasks demonstrate that these methods can effectively improve image restoration performance.
  arXiv  Detail & Related papers  (2023-09-21T01:35:13Z)
• SILT: Shadow-aware Iterative Label Tuning for Learning to Detect Shadows from Noisy Labels [53.30604926018168]
  We propose SILT, the Shadow-aware Iterative Label Tuning framework, which explicitly considers noise in shadow labels and trains the deep model in a self-training manner. We also devise a simple yet effective label tuning strategy with global-local fusion and shadow-aware filtering to encourage the network to make significant refinements on the noisy labels. Our results show that even a simple U-Net trained with SILT can outperform all state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2023-08-23T11:16:36Z)
• ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal [74.86415440438051]
  We propose a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. Our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.
  arXiv  Detail & Related papers  (2022-12-09T07:48:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.