Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling

• URL: http://arxiv.org/abs/2306.17358v3
• Date: Thu, 4 Jan 2024 09:00:11 GMT
• Title: Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling
• Authors: Xinhao Tao, Junyan Cao, Yan Hong, Li Niu
• Abstract summary: We focus on generating plausible shadows for the inserted foreground object to make the composite image more realistic. To supplement the existing small-scale dataset, we create a large-scale dataset called RdSOBA with rendering techniques. We design a two-stage network named DMASNet with mask prediction and attentive shadow filling.
• Score: 26.780859992812186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image composition refers to inserting a foreground object into a background image to obtain a composite image. In this work, we focus on generating plausible shadows for the inserted foreground object to make the composite image more realistic. To supplement the existing small-scale dataset, we create a large-scale dataset called RdSOBA with rendering techniques. Moreover, we design a two-stage network named DMASNet with decomposed mask prediction and attentive shadow filling. Specifically, in the first stage, we decompose shadow mask prediction into box prediction and shape prediction. In the second stage, we attend to reference background shadow pixels to fill the foreground shadow. Abundant experiments prove that our DMASNet achieves better visual effects and generalizes well to real composite images.

Related papers

• SoftShadow: Leveraging Penumbra-Aware Soft Masks for Shadow Removal [35.16957947180504]
  We introduce novel soft shadow masks specifically designed for shadow removal. To achieve such soft masks, we propose a textitSoftShadow framework by leveraging the prior knowledge of pretrained SAM. This framework enables accurate predictions of penumbra (partially shaded regions) and umbra (fully shaded regions) areas while simultaneously facilitating end-to-end shadow removal.
  arXiv  Detail & Related papers  (2024-09-11T06:12:26Z)
• Shadow Generation for Composite Image Using Diffusion model [16.316311264197324]
  We resort to foundation model with rich prior knowledge of natural shadow images. We first adapt ControlNet to our task and then propose intensity modulation modules to improve the shadow intensity. Experimental results on both DESOBA and DESOBAv2 datasets as well as real composite images demonstrate the superior capability of our model for shadow generation task.
  arXiv  Detail & Related papers  (2024-03-22T14:27:58Z)
• Seeing the Unseen: Visual Common Sense for Semantic Placement [71.76026880991245]
  Given an image, a vision system is asked to predict semantically-meaningful regions (masks or bounding boxes) in the image where that object could be placed or is likely be placed by humans. We call this task: Semantic Placement (SP) and believe that such common-sense visual understanding is critical for assitive robots (tidying a house), and AR devices (automatically rendering an object in the user's space)
  arXiv  Detail & Related papers  (2024-01-15T15:28:30Z)
• DESOBAv2: Towards Large-scale Real-world Dataset for Shadow Generation [19.376935979734714]
  In this work, we focus on generating plausible shadow for the inserted foreground object to make the composite image more realistic. To supplement the existing small-scale dataset DESOBA, we create a large-scale dataset called DESOBAv2.
  arXiv  Detail & Related papers  (2023-08-19T10:21:23Z)
• Shadow Removal by High-Quality Shadow Synthesis [78.56549207362863]
  HQSS employs a shadow feature encoder and a generator to synthesize pseudo images. HQSS is observed to outperform the state-of-the-art methods on ISTD dataset, Video Shadow Removal dataset, and SRD dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:52:52Z)
• Controllable Shadow Generation Using Pixel Height Maps [58.59256060452418]
  Physics-based shadow rendering methods require 3D geometries, which are not always available. Deep learning-based shadow synthesis methods learn a mapping from the light information to an object's shadow without explicitly modeling the shadow geometry. We introduce pixel heigh, a novel geometry representation that encodes the correlations between objects, ground, and camera pose.
  arXiv  Detail & Related papers  (2022-07-12T08:29:51Z)
• Layered Depth Refinement with Mask Guidance [61.10654666344419]
  We formulate a novel problem of mask-guided depth refinement that utilizes a generic mask to refine the depth prediction of SIDE models. Our framework performs layered refinement and inpainting/outpainting, decomposing the depth map into two separate layers signified by the mask and the inverse mask. We empirically show that our method is robust to different types of masks and initial depth predictions, accurately refining depth values in inner and outer mask boundary regions.
  arXiv  Detail & Related papers  (2022-06-07T06:42:44Z)
• SIDNet: Learning Shading-aware Illumination Descriptor for Image Harmonization [10.655037947250516]
  Image harmonization aims at adjusting the appearance of the foreground to make it more compatible with the background. We decompose the image harmonization task into two sub-problems: 1) illumination estimation of the background image and 2) re-rendering of foreground objects under background illumination.
  arXiv  Detail & Related papers  (2021-12-02T15:18:29Z)
• Shadow Generation for Composite Image in Real-world Scenes [23.532079444113528]
  We propose a novel shadow generation network SGRNet, which consists of a shadow mask prediction stage and a shadow filling stage. In the shadow mask prediction stage, foreground and background information are thoroughly interacted to generate foreground shadow mask. In the shadow filling stage, shadow parameters are predicted to fill the shadow area.
  arXiv  Detail & Related papers  (2021-04-21T03:30:02Z)
• Adversarial Image Composition with Auxiliary Illumination [53.89445873577062]
  We propose an Adversarial Image Composition Net (AIC-Net) that achieves realistic image composition. A novel branched generation mechanism is proposed, which disentangles the generation of shadows and the transfer of foreground styles. Experiments on pedestrian and car composition tasks show that the proposed AIC-Net achieves superior composition performance.
  arXiv  Detail & Related papers  (2020-09-17T12:58:16Z)
• BachGAN: High-Resolution Image Synthesis from Salient Object Layout [78.51640906030244]
  We propose a new task towards more practical application for image generation - high-quality image synthesis from salient object layout. Two main challenges spring from this new task: (i) how to generate fine-grained details and realistic textures without segmentation map input; and (ii) how to create a background and weave it seamlessly into standalone objects. By generating the hallucinated background representation dynamically, our model can synthesize high-resolution images with both photo-realistic foreground and integral background.
  arXiv  Detail & Related papers  (2020-03-26T00:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.