Prompt-Aware Controllable Shadow Removal

• URL: http://arxiv.org/abs/2501.15043v2
• Date: Mon, 03 Feb 2025 03:30:09 GMT
• Title: Prompt-Aware Controllable Shadow Removal
• Authors: Kerui Chen, Zhiliang Wu, Wenjin Hou, Kun Li, Hehe Fan, Yi Yang,
• Abstract summary: We introduce a novel paradigm for prompt-aware controllable shadow removal. Unlike existing approaches, our paradigm allows for targeted shadow removal from specific subjects based on user prompts. We propose an end-to-end learnable model, the Prompt-Aware Controllable Shadow Removal Network (PACSRNet)
• Score: 29.674151621173856
• License:
• Abstract: Shadow removal aims to restore the image content in shadowed regions. While deep learning-based methods have shown promising results, they still face key challenges: 1) uncontrolled removal of all shadows, or 2) controllable removal but heavily relies on precise shadow region masks. To address these issues, we introduce a novel paradigm: prompt-aware controllable shadow removal. Unlike existing approaches, our paradigm allows for targeted shadow removal from specific subjects based on user prompts (e.g., dots, lines, or subject masks). This approach eliminates the need for shadow annotations and offers flexible, user-controlled shadow removal. Specifically, we propose an end-to-end learnable model, the Prompt-Aware Controllable Shadow Removal Network (PACSRNet). PACSRNet consists of two key modules: a prompt-aware module that generates shadow masks for the specified subject based on the user prompt, and a shadow removal module that uses the shadow prior from the first module to restore the content in the shadowed regions. Additionally, we enhance the shadow removal module by incorporating feature information from the prompt-aware module through a linear operation, providing prompt-guided support for shadow removal. Recognizing that existing shadow removal datasets lack diverse user prompts, we contribute a new dataset specifically designed for prompt-based controllable shadow removal. Extensive experimental results demonstrate the effectiveness and superiority of PACSRNet.

Related papers

• MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis [64.00425120075045]
  Shadows are often under-considered or even ignored in image editing applications, limiting the realism of the edited results. In this paper, we introduce MetaShadow, a three-in-one versatile framework that enables detection, removal, and controllable synthesis of shadows in natural images in an object-centered fashion.
  arXiv  Detail & Related papers  (2024-12-03T18:04:42Z)
• SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal [35.16957947180504]
  We introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a textitSoftShadow framework by leveraging the prior knowledge of pretrained SAM. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal.
  arXiv  Detail & Related papers  (2024-09-11T06:12:26Z)
• Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey [78.84004293081631]
  The patterns of shadows are arbitrary, varied, and often have highly complex trace structures. The degradation caused by shadows is spatially non-uniform, resulting in inconsistencies in illumination and color between shadow and non-shadow areas. Recent developments in this field are primarily driven by deep learning-based solutions.
  arXiv  Detail & Related papers  (2024-07-11T20:58:38Z)
• Progressive Recurrent Network for Shadow Removal [99.1928825224358]
  Single-image shadow removal is a significant task that is still unresolved. Most existing deep learning-based approaches attempt to remove the shadow directly, which can not deal with the shadow well. We propose a simple but effective Progressive Recurrent Network (PRNet) to remove the shadow progressively.
  arXiv  Detail & Related papers  (2023-11-01T11:42:45Z)
• Structure-Informed Shadow Removal Networks [67.57092870994029]
  Existing deep learning-based shadow removal methods still produce images with shadow remnants. We propose a novel structure-informed shadow removal network (StructNet) to leverage the image-structure information to address the shadow remnant problem. Our method outperforms existing shadow removal methods, and our StructNet can be integrated with existing methods to improve them further.
  arXiv  Detail & Related papers  (2023-01-09T06:31:52Z)
• CNSNet: A Cleanness-Navigated-Shadow Network for Shadow Removal [4.951051823391577]
  We propose a shadow-oriented adaptive normalization (SOAN) module and a shadow-aware aggregation with transformer (SAAT) module based on the shadow mask. Under the guidance of the shadow mask, the SOAN module formulates the statistics from the non-shadow region and adaptively applies them to the shadow region for region-wise restoration. The SAAT module utilizes the shadow mask to precisely guide the restoration of each shadowed pixel by considering the highly relevant pixels from the shadow-free regions for global pixel-wise restoration.
  arXiv  Detail & Related papers  (2022-09-06T01:33:38Z)
• DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network [28.6541488555978]
  We propose an unsupervised domain-classifier guided shadow removal network, DC-ShadowNet. We introduce novel losses based on physics-based shadow-free chromaticity, shadow-robust perceptual features, and boundary smoothness. Our experiments show that all these novel components allow our method to handle soft shadows, and also to perform better on hard shadows.
  arXiv  Detail & Related papers  (2022-07-21T12:04:16Z)
• UnShadowNet: Illumination Critic Guided Contrastive Learning For Shadow Removal [14.898039056038789]
  We introduce a novel weakly supervised shadow removal framework UnShadowNet. It is composed of a DeShadower network responsible for the removal of the extracted shadow under the guidance of an Illumination network. We show that UnShadowNet can be easily extended to a fully-supervised set-up to exploit the ground-truth when available.
  arXiv  Detail & Related papers  (2022-03-29T11:17:02Z)
• R2D: Learning Shadow Removal to Enhance Fine-Context Shadow Detection [64.10636296274168]
  Current shadow detection methods perform poorly when detecting shadow regions that are small, unclear or have blurry edges. We propose a new method called Restore to Detect (R2D), where a deep neural network is trained for restoration (shadow removal) We show that our proposed method R2D improves the shadow detection performance while being able to detect fine context better compared to the other recent methods.
  arXiv  Detail & Related papers  (2021-09-20T15:09:22Z)
• Self-Supervised Shadow Removal [130.6657167667636]
  We propose an unsupervised single image shadow removal solution via self-supervised learning by using a conditioned mask. In contrast to existing literature, we do not require paired shadowed and shadow-free images, instead we rely on self-supervision and jointly learn deep models to remove and add shadows to images.
  arXiv  Detail & Related papers  (2020-10-22T11:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.