BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

• URL: http://arxiv.org/abs/2112.06179v1
• Date: Sun, 12 Dec 2021 08:20:01 GMT
• Title: BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning
• Authors: Changgyoon Oh, Wonjune Cho, Daehee Park, Yujeong Chae, Lin Wang and Kuk-Jin Yoon
• Abstract summary: We propose a novel bi-modal (RGB-D) panorama synthesis framework. We focus on indoor environments where the RGB-D panorama can provide a complete 3D model for many applications. Our method synthesizes high-quality indoor RGB-D panoramas and provides realistic 3D indoor models.
• Score: 26.24526760567159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Providing omnidirectional depth along with RGB information is important for numerous applications, eg, VR/AR. However, as omnidirectional RGB-D data is not always available, synthesizing RGB-D panorama data from limited information of a scene can be useful. Therefore, some prior works tried to synthesize RGB panorama images from perspective RGB images; however, they suffer from limited image quality and can not be directly extended for RGB-D panorama synthesis. In this paper, we study a new problem: RGB-D panorama synthesis under the arbitrary configurations of cameras and depth sensors. Accordingly, we propose a novel bi-modal (RGB-D) panorama synthesis (BIPS) framework. Especially, we focus on indoor environments where the RGB-D panorama can provide a complete 3D model for many applications. We design a generator that fuses the bi-modal information and train it with residual-aided adversarial learning (RDAL). RDAL allows to synthesize realistic indoor layout structures and interiors by jointly inferring RGB panorama, layout depth, and residual depth. In addition, as there is no tailored evaluation metric for RGB-D panorama synthesis, we propose a novel metric to effectively evaluate its perceptual quality. Extensive experiments show that our method synthesizes high-quality indoor RGB-D panoramas and provides realistic 3D indoor models than prior methods. Code will be released upon acceptance.

Related papers

• PERF: Panoramic Neural Radiance Field from a Single Panorama [109.31072618058043]
  PERF is a novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. We propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.
  arXiv  Detail & Related papers  (2023-10-25T17:59:01Z)
• A novel approach for holographic 3D content generation without depth map [2.905273049932301]
  We propose a deep learning-based method to synthesize the volumetric digital holograms using only the given RGB image. Through experiments, we demonstrate that the volumetric hologram generated through our proposed model is more accurate than that of competitive models.
  arXiv  Detail & Related papers  (2023-09-26T14:37:31Z)
• PanoDiffusion: 360-degree Panorama Outpainting via Diffusion [29.10215003547589]
  We present our 360-degree indoor RGB-D panorama outpainting model using latent diffusion models (LDM), called PanoDiffusion. We introduce a new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images during inference. Results show that our PanoDiffusion not only significantly outperforms state-of-the-art methods on RGB-D panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-quality depth panoramas to provide realistic 3D indoor models.
  arXiv  Detail & Related papers  (2023-07-06T17:57:02Z)
• Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution [123.04455334124188]
  Guided depth map super-resolution (GDSR) aims to upsample low-resolution (LR) depth maps with additional information involved in high-resolution (HR) RGB images from the same scene. In this paper, we propose the Spherical Space feature Decomposition Network (SSDNet) to solve the above issues. Our method can achieve state-of-the-art results on four test datasets, as well as successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2023-03-15T21:22:21Z)
• Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision [51.385731364529306]
  This paper focuses on perceiving and navigating 3D environments using echoes and RGB image. In particular, we perform depth estimation by fusing RGB image with echoes, received from multiple orientations. We show that the echoes provide holistic and in-expensive information about the 3D structures complementing the RGB image.
  arXiv  Detail & Related papers  (2022-07-03T22:31:47Z)
• Urban Radiance Fields [77.43604458481637]
  We perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings. Each of these three extensions provides significant performance improvements in experiments on Street View data.
  arXiv  Detail & Related papers  (2021-11-29T15:58:16Z)
• Deep Multi Depth Panoramas for View Synthesis [70.9125433400375]
  We present a novel scene representation - Multi Depth Panorama (MDP) - that consists of multiple RGBD$alpha$ panoramas. MDPs are more compact than previous 3D scene representations and enable high-quality, efficient new view rendering.
  arXiv  Detail & Related papers  (2020-08-04T20:29:15Z)
• NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image [34.79657678041356]
  We propose a fast adversarial learning-based method to reconstruct the complete and detailed 3D human from a single RGB-D image. Given a consumer RGB-D sensor, NormalGAN can generate the complete and detailed 3D human reconstruction results in 20 fps.
  arXiv  Detail & Related papers  (2020-07-30T09:35:46Z)
• Synergistic saliency and depth prediction for RGB-D saliency detection [76.27406945671379]
  Existing RGB-D saliency datasets are small, which may lead to overfitting and limited generalization for diverse scenarios. We propose a semi-supervised system for RGB-D saliency detection that can be trained on smaller RGB-D saliency datasets without saliency ground truth.
  arXiv  Detail & Related papers  (2020-07-03T14:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.