ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal

• URL: http://arxiv.org/abs/2404.18433v2
• Date: Tue, 30 Apr 2024 15:42:25 GMT
• Title: ShadowMaskFormer: Mask Augmented Patch Embeddings for Shadow Removal
• Authors: Zhuohao Li, Guoyang Xie, Guannan Jiang, Zhichao Lu,
• Abstract summary: We propose a transformer-based framework with a novel patch embedding that is tailored for shadow removal, dubbed ShadowMaskFormer. Specifically, we present a simple and effective mask-augmented patch embedding to integrate shadow information and promote the model's emphasis on acquiring knowledge for shadow regions.
• Score: 13.983288991595614
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Transformer recently emerged as the de facto model for computer vision tasks and has also been successfully applied to shadow removal. However, these existing methods heavily rely on intricate modifications to the attention mechanisms within the transformer blocks while using a generic patch embedding. As a result, it often leads to complex architectural designs requiring additional computation resources. In this work, we aim to explore the efficacy of incorporating shadow information within the early processing stage. Accordingly, we propose a transformer-based framework with a novel patch embedding that is tailored for shadow removal, dubbed ShadowMaskFormer. Specifically, we present a simple and effective mask-augmented patch embedding to integrate shadow information and promote the model's emphasis on acquiring knowledge for shadow regions. Extensive experiments conducted on the ISTD, ISTD+, and SRD benchmark datasets demonstrate the efficacy of our method against state-of-the-art approaches while using fewer model parameters.

Related papers

• Towards Hard and Soft Shadow Removal via Dual-Branch Separation Network and Vision Transformer [0.0]
  We propose a dual-path model that processes hard and soft shadows separately. The model classifies shadow types and processes them through appropriate paths to produce shadow-free outputs. Our model outperforms state-of-the-art methods and achieves 2.905 RMSE value on the ISTD dataset.
  arXiv  Detail & Related papers  (2025-01-03T15:29:46Z)
• MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis [64.00425120075045]
  Shadows are often under-considered or even ignored in image editing applications, limiting the realism of the edited results. In this paper, we introduce MetaShadow, a three-in-one versatile framework that enables detection, removal, and controllable synthesis of shadows in natural images in an object-centered fashion.
  arXiv  Detail & Related papers  (2024-12-03T18:04:42Z)
• SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal [35.16957947180504]
  We introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a textitSoftShadow framework by leveraging the prior knowledge of pretrained SAM. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal.
  arXiv  Detail & Related papers  (2024-09-11T06:12:26Z)
• SwinShadow: Shifted Window for Ambiguous Adjacent Shadow Detection [90.4751446041017]
  We present SwinShadow, a transformer-based architecture that fully utilizes the powerful shifted window mechanism for detecting adjacent shadows. The whole process can be divided into three parts: encoder, decoder, and feature integration. Experiments on three shadow detection benchmark datasets, SBU, UCF, and ISTD, demonstrate that our network achieves good performance in terms of balance error rate (BER)
  arXiv  Detail & Related papers  (2024-08-07T03:16:33Z)
• ShaDocFormer: A Shadow-Attentive Threshold Detector With Cascaded Fusion Refiner for Document Shadow Removal [26.15238399758745]
  We propose a Transformer-based architecture that integrates traditional methodologies and deep learning techniques to tackle the problem of document shadow removal. The ShaDocFormer architecture comprises two components: the Shadow-attentive Threshold Detector (STD) and the Cascaded Fusion Refiner (CFR)
  arXiv  Detail & Related papers  (2023-09-13T02:15:29Z)
• ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal [74.86415440438051]
  We propose a unified diffusion framework that integrates both the image and degradation priors for highly effective shadow removal. Our model achieves a significant improvement in terms of PSNR, increasing from 31.69dB to 34.73dB over SRD dataset.
  arXiv  Detail & Related papers  (2022-12-09T07:48:30Z)
• ShaDocNet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removal [53.01990632289937]
  We propose a Transformer-based model for document shadow removal. It uses shadow context encoding and decoding in both shadow and shadow-free regions.
  arXiv  Detail & Related papers  (2022-11-30T01:46:29Z)
• CNSNet: A Cleanness-Navigated-Shadow Network for Shadow Removal [4.951051823391577]
  We propose a shadow-oriented adaptive normalization (SOAN) module and a shadow-aware aggregation with transformer (SAAT) module based on the shadow mask. Under the guidance of the shadow mask, the SOAN module formulates the statistics from the non-shadow region and adaptively applies them to the shadow region for region-wise restoration. The SAAT module utilizes the shadow mask to precisely guide the restoration of each shadowed pixel by considering the highly relevant pixels from the shadow-free regions for global pixel-wise restoration.
  arXiv  Detail & Related papers  (2022-09-06T01:33:38Z)
• MAT: Mask-Aware Transformer for Large Hole Image Inpainting [79.67039090195527]
  We present a novel model for large hole inpainting, which unifies the merits of transformers and convolutions. Experiments demonstrate the state-of-the-art performance of the new model on multiple benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T06:36:17Z)
• Self-Supervised Shadow Removal [130.6657167667636]
  We propose an unsupervised single image shadow removal solution via self-supervised learning by using a conditioned mask. In contrast to existing literature, we do not require paired shadowed and shadow-free images, instead we rely on self-supervision and jointly learn deep models to remove and add shadows to images.
  arXiv  Detail & Related papers  (2020-10-22T11:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.