SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal

• URL: http://arxiv.org/abs/2409.07041v1
• Date: Wed, 11 Sep 2024 06:12:26 GMT
• Title: SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal
• Authors: Xinrui Wang, Lanqing Guo, Xiyu Wang, Siyu Huang, Bihan Wen,
• Abstract summary: We introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a textitSoftShadow framework by leveraging the prior knowledge of pretrained SAM. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal.
• Score: 35.16957947180504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in deep learning have yielded promising results for the image shadow removal task. However, most existing methods rely on binary pre-generated shadow masks. The binary nature of such masks could potentially lead to artifacts near the boundary between shadow and non-shadow areas. In view of this, inspired by the physical model of shadow formation, we introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a \textit{SoftShadow} framework by leveraging the prior knowledge of pretrained SAM and integrating physical constraints. Specifically, we jointly tune the SAM and the subsequent shadow removal network using penumbra formation constraint loss and shadow removal loss. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal. Through extensive experiments on popular datasets, we found that our SoftShadow framework, which generates soft masks, can better restore boundary artifacts, achieve state-of-the-art performance, and demonstrate superior generalizability.

Related papers

• Shadow Removal Refinement via Material-Consistent Shadow Edges [33.8383848078524]
  On both sides of shadow edges traversing regions with the same material, the original color and textures should be the same if the shadow is removed properly. We fine-tune SAM, an image segmentation foundation model, to produce a shadow-invariant segmentation and then extract material-consistent shadow edges. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging, in-the-wild images.
  arXiv  Detail & Related papers  (2024-09-10T20:16:28Z)
• Shadow Removal by High-Quality Shadow Synthesis [78.56549207362863]
  HQSS employs a shadow feature encoder and a generator to synthesize pseudo images. HQSS is observed to outperform the state-of-the-art methods on ISTD dataset, Video Shadow Removal dataset, and SRD dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:52:52Z)
• DeS3: Adaptive Attention-driven Self and Soft Shadow Removal using ViT Similarity [54.831083157152136]
  We present a method that removes hard, soft and self shadows based on adaptive attention and ViT similarity. Our method outperforms state-of-the-art methods on the SRD, AISTD, LRSS, USR and UIUC datasets.
  arXiv  Detail & Related papers  (2022-11-15T12:15:29Z)
• Shadow-Aware Dynamic Convolution for Shadow Removal [80.82708225269684]
  We introduce a novel Shadow-Aware Dynamic Convolution (SADC) module to decouple the interdependence between the shadow region and the non-shadow region. Inspired by the fact that the color mapping of the non-shadow region is easier to learn, our SADC processes the non-shadow region with a lightweight convolution module. We develop a novel intra-convolution distillation loss to strengthen the information flow from the non-shadow region to the shadow region.
  arXiv  Detail & Related papers  (2022-05-10T14:00:48Z)
• R2D: Learning Shadow Removal to Enhance Fine-Context Shadow Detection [64.10636296274168]
  Current shadow detection methods perform poorly when detecting shadow regions that are small, unclear or have blurry edges. We propose a new method called Restore to Detect (R2D), where a deep neural network is trained for restoration (shadow removal) We show that our proposed method R2D improves the shadow detection performance while being able to detect fine context better compared to the other recent methods.
  arXiv  Detail & Related papers  (2021-09-20T15:09:22Z)
• From Shadow Generation to Shadow Removal [19.486543304598264]
  We propose a new G2R-ShadowNet which leverages shadow generation for weakly-supervised shadow removal. The proposed G2R-ShadowNet consists of three sub-networks for shadow generation, shadow removal and refinement. In particular, the shadow generation sub-net stylises non-shadow regions to be shadow ones, leading to paired data for training the shadow-removal sub-net.
  arXiv  Detail & Related papers  (2021-03-24T05:49:08Z)
• Learning from Synthetic Shadows for Shadow Detection and Removal [43.53464469097872]
  Recent shadow removal approaches all train convolutional neural networks (CNN) on real paired shadow/shadow-free or shadow/shadow-free/mask image datasets. We present SynShadow, a novel large-scale synthetic shadow/shadow-free/matte image triplets dataset and a pipeline to synthesize it.
  arXiv  Detail & Related papers  (2021-01-05T18:56:34Z)
• Physics-based Shadow Image Decomposition for Shadow Removal [36.41558227710456]
  We propose a novel deep learning method for shadow removal. Inspired by physical models of shadow formation, we use a linear illumination transformation to model the shadow effects in the image. We train and test our framework on the most challenging shadow removal dataset.
  arXiv  Detail & Related papers  (2020-12-23T23:06:38Z)
• Self-Supervised Shadow Removal [130.6657167667636]
  We propose an unsupervised single image shadow removal solution via self-supervised learning by using a conditioned mask. In contrast to existing literature, we do not require paired shadowed and shadow-free images, instead we rely on self-supervision and jointly learn deep models to remove and add shadows to images.
  arXiv  Detail & Related papers  (2020-10-22T11:33:41Z)
• From Shadow Segmentation to Shadow Removal [34.762493656937366]
  The requirement for paired shadow and shadow-free images limits the size and diversity of shadow removal datasets. We propose a shadow removal method that can be trained using only shadow and non-shadow patches cropped from the shadow images themselves.
  arXiv  Detail & Related papers  (2020-08-01T14:00:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.