HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined RGB and Depth Inpainting

• URL: http://arxiv.org/abs/2402.10334v1
• Date: Thu, 15 Feb 2024 21:43:56 GMT
• Title: HI-GAN: Hierarchical Inpainting GAN with Auxiliary Inputs for Combined RGB and Depth Inpainting
• Authors: Ankan Dash, Jingyi Gu and Guiling Wang
• Abstract summary: Inpainting involves filling in missing pixels or areas in an image. Existing methods rely on digital replacement techniques which necessitate multiple cameras and incur high costs. We propose Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in a hierarchical fashion for RGBD inpainting.
• Score: 3.736916304884176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inpainting involves filling in missing pixels or areas in an image, a crucial technique employed in Mixed Reality environments for various applications, particularly in Diminished Reality (DR) where content is removed from a user's visual environment. Existing methods rely on digital replacement techniques which necessitate multiple cameras and incur high costs. AR devices and smartphones use ToF depth sensors to capture scene depth maps aligned with RGB images. Despite speed and affordability, ToF cameras create imperfect depth maps with missing pixels. To address the above challenges, we propose Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked edge and segmentation label images respectively, while CombinedRGBD-GAN combines their latent representation outputs and performs RGB and Depth inpainting. Edge images and particularly segmentation label images as auxiliary inputs significantly enhance inpainting performance by complementary context and hierarchical optimization. We believe we make the first attempt to incorporate label images into inpainting process.Unlike previous approaches requiring multiple sequential models and separate outputs, our work operates in an end-to-end manner, training all three models simultaneously and hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized separately and further optimized inside CombinedRGBD-GAN to enhance inpainting quality. Experiments demonstrate that HI-GAN works seamlessly and achieves overall superior performance compared with existing approaches.

Related papers

• MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior [65.05773512126089]
  NeRF inpainting methods built upon explicit RGB and depth 2D inpainting supervisions are inherently constrained by the capabilities of their underlying 2D inpainters. We propose MVIP-NeRF that harnesses the potential of diffusion priors for NeRF inpainting, addressing both appearance and geometry aspects. Our experimental results show better appearance and geometry recovery than previous NeRF inpainting methods.
  arXiv  Detail & Related papers  (2024-05-05T09:04:42Z)
• FDCE-Net: Underwater Image Enhancement with Embedding Frequency and Dual Color Encoder [49.79611204954311]
  Underwater images often suffer from various issues such as low brightness, color shift, blurred details, and noise due to absorption light and scattering caused by water and suspended particles. Previous underwater image enhancement (UIE) methods have primarily focused on spatial domain enhancement, neglecting the frequency domain information inherent in the images.
  arXiv  Detail & Related papers  (2024-04-27T15:16:34Z)
• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models [59.01600111737628]
  HD-Painter is a training free approach that accurately follows prompts and coherently scales to high resolution image inpainting. To this end, we design the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention scores. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-12-21T18:09:30Z)
• UniGS: Unified Representation for Image Generation and Segmentation [105.08152635402858]
  We use a colormap to represent entity-level masks, addressing the challenge of varying entity numbers. Two novel modules, including the location-aware color palette and progressive dichotomy module, are proposed to support our mask representation.
  arXiv  Detail & Related papers  (2023-12-04T15:59:27Z)
• High-Fidelity Image Inpainting with GAN Inversion [23.49170140410603]
  In this paper, we propose a novel GAN inversion model for image inpainting, dubbed InvertFill. Within the encoder, the pre-modulation network leverages multi-scale structures to encode more discriminative semantics into style vectors. To reconstruct faithful and photorealistic images, a simple yet effective Soft-update Mean Latent module is designed to capture more diverse in-domain patterns that synthesize high-fidelity textures for large corruptions.
  arXiv  Detail & Related papers  (2022-08-25T03:39:24Z)
• MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting [35.79101039727397]
  We study the advantages and challenges of image-level predictive filtering for image inpainting. We propose a novel filtering technique, i.e., Multilevel Interactive Siamese Filtering (MISF), which contains two branches: kernel prediction branch (KPB) and semantic & image filtering branch (SIFB) Our method outperforms state-of-the-art baselines on four metrics, i.e., L1, PSNR, SSIM, and LPIPS.
  arXiv  Detail & Related papers  (2022-03-12T01:32:39Z)
• RSINet: Inpainting Remotely Sensed Images Using Triple GAN Framework [13.613245876782367]
  We propose a novel inpainting method that individually focuses on each aspect of an image such as edges, colour and texture. Each individual GAN also incorporates the attention mechanism that explicitly extracts the spectral and spatial features. We evaluate our model, alongwith previous state of the art models, on the two well known remote sensing datasets, Open Cities AI and Earth on Canvas.
  arXiv  Detail & Related papers  (2022-02-12T05:19:37Z)
• Aggregated Contextual Transformations for High-Resolution Image Inpainting [57.241749273816374]
  We propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN) for high-resolution image inpainting. To enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task.
  arXiv  Detail & Related papers  (2021-04-03T15:50:17Z)
• 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
  We propose a method for converting a single RGB-D input image into a 3D photo. A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region. The resulting 3D photos can be efficiently rendered with motion parallax.
  arXiv  Detail & Related papers  (2020-04-09T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.