Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models

• URL: http://arxiv.org/abs/2212.05993v1
• Date: Mon, 12 Dec 2022 15:50:00 GMT
• Title: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models
• Authors: Jiabao Lei, Jiapeng Tang, Kui Jia
• Abstract summary: In this work, we present a new solution that sequentially generates novel RGBD views along a camera trajectory. Each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the refractory problem of multi-view inconsistency.
• Score: 39.23531919945332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the refractory problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/project-pages/rgbd-diffusion.html

Related papers

• GenRC: Generative 3D Room Completion from Sparse Image Collections [17.222652213723485]
  GenRC is an automated training-free pipeline to complete a room-scale 3D mesh with high-fidelity textures. E-Diffusion generates a view-consistent panoramic RGBD image which ensures global geometry and appearance consistency. GenRC outperforms state-of-the-art methods under most appearance and geometric metrics on ScanNet and ARKitScenes datasets.
  arXiv  Detail & Related papers  (2024-07-17T18:10:40Z)
• MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation [54.27399121779011]
  We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images. We show that our approach can yield more accurate synthesis compared to recent state-of-the-art, including distillation-based 3D inference and prior multi-view generation methods.
  arXiv  Detail & Related papers  (2024-04-04T17:59:57Z)
• O$^2$-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model [28.372289119872764]
  Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects. We propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects.
  arXiv  Detail & Related papers  (2023-08-18T14:38:31Z)
• Color-NeuS: Reconstructing Neural Implicit Surfaces with Color [46.90825914361547]
  We develop a method to reconstruct object surfaces from multi-view images or monocular video. We remove the view-dependent color from neural volume rendering while retaining volume rendering performance through a relighting network. Results surpass those of any existing methods capable of reconstructing mesh alongside color.
  arXiv  Detail & Related papers  (2023-08-14T06:32:54Z)
• $PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction [97.06927852165464]
  Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. We propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process.
  arXiv  Detail & Related papers  (2023-02-21T13:37:07Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• RGB-D Image Inpainting Using Generative Adversarial Network with a Late Fusion Approach [14.06830052027649]
  Diminished reality is a technology that aims to remove objects from video images and fills in the missing region with plausible pixels. We propose an RGB-D image inpainting method using generative adversarial network.
  arXiv  Detail & Related papers  (2021-10-14T14:44:01Z)
• Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images [69.5662419067878]
  Grounding referring expressions in RGBD image has been an emerging field. We present a novel task of 3D visual grounding in single-view RGBD image where the referred objects are often only partially scanned due to occlusion. Our approach first fuses the language and the visual features at the bottom level to generate a heatmap that localizes the relevant regions in the RGBD image. Then our approach conducts an adaptive feature learning based on the heatmap and performs the object-level matching with another visio-linguistic fusion to finally ground the referred object.
  arXiv  Detail & Related papers  (2021-03-14T11:18:50Z)
• RGBD-Net: Predicting color and depth images for novel views synthesis [46.233701784858184]
  RGBD-Net is proposed to predict the depth map and the color images at the target pose in a multi-scale manner. The results indicate that RGBD-Net generalizes well to previously unseen data.
  arXiv  Detail & Related papers  (2020-11-29T16:42:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.