Behind the Scenes: Density Fields for Single View Reconstruction

• URL: http://arxiv.org/abs/2301.07668v3
• Date: Wed, 19 Apr 2023 15:01:39 GMT
• Title: Behind the Scenes: Density Fields for Single View Reconstruction
• Authors: Felix Wimbauer, Nan Yang, Christian Rupprecht, Daniel Cremers
• Abstract summary: Inferring meaningful geometric scene representation from a single image is a fundamental problem in computer vision. We propose to predict implicit density fields. A density field maps every location in the frustum of the input image to volumetric density. We show that our method is able to predict meaningful geometry for regions that are occluded in the input image.
• Score: 63.40484647325238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inferring a meaningful geometric scene representation from a single image is a fundamental problem in computer vision. Approaches based on traditional depth map prediction can only reason about areas that are visible in the image. Currently, neural radiance fields (NeRFs) can capture true 3D including color, but are too complex to be generated from a single image. As an alternative, we propose to predict implicit density fields. A density field maps every location in the frustum of the input image to volumetric density. By directly sampling color from the available views instead of storing color in the density field, our scene representation becomes significantly less complex compared to NeRFs, and a neural network can predict it in a single forward pass. The prediction network is trained through self-supervision from only video data. Our formulation allows volume rendering to perform both depth prediction and novel view synthesis. Through experiments, we show that our method is able to predict meaningful geometry for regions that are occluded in the input image. Additionally, we demonstrate the potential of our approach on three datasets for depth prediction and novel-view synthesis.

Related papers

• Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation [39.08243715525956]
  Inferring scene geometry from images via Structure from Motion is a long-standing and fundamental problem in computer vision. With the popularity of neural radiance fields (NeRFs), implicit representations also became popular for scene completion. We propose to fuse the scene reconstruction from multiple images and distill this knowledge into a more accurate single-view scene reconstruction.
  arXiv  Detail & Related papers  (2024-04-11T17:30:24Z)
• Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning [119.99066522299309]
  KYN is a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work.
  arXiv  Detail & Related papers  (2024-04-04T17:59:59Z)
• Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors [32.63878457242185]
  We learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene. Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods.
  arXiv  Detail & Related papers  (2023-10-17T21:45:51Z)
• One-Shot Neural Fields for 3D Object Understanding [112.32255680399399]
  We present a unified and compact scene representation for robotics. Each object in the scene is depicted by a latent code capturing geometry and appearance. This representation can be decoded for various tasks such as novel view rendering, 3D reconstruction, and stable grasp prediction.
  arXiv  Detail & Related papers  (2022-10-21T17:33:14Z)
• S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint [22.42916940712357]
  Our method learns a neural reflectance field to represent the 3D geometry and BRDFs of a scene. Our method is capable of recovering 3D geometry, including both visible and invisible parts, of a scene from single-view images. It supports applications like novel-view synthesis and relighting.
  arXiv  Detail & Related papers  (2022-10-17T11:01:52Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering [60.02806355570514]
  Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. We propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field. Rendering a ray from an LFN requires only a *single* network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or based on volumetrics.
  arXiv  Detail & Related papers  (2021-06-04T17:54:49Z)
• NeMI: Unifying Neural Radiance Fields with Multiplane Images for Novel View Synthesis [69.19261797333635]
  We propose an approach to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our NeMI unifies Neural radiance fields (NeRF) with Multiplane Images (MPI) We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision.
  arXiv  Detail & Related papers  (2021-03-27T13:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.