Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation

• URL: http://arxiv.org/abs/2404.07933v1
• Date: Thu, 11 Apr 2024 17:30:24 GMT
• Title: Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation
• Authors: Keonhee Han, Dominik Muhle, Felix Wimbauer, Daniel Cremers,
• Abstract summary: Inferring scene geometry from images via Structure from Motion is a long-standing and fundamental problem in computer vision. With the popularity of neural radiance fields (NeRFs), implicit representations also became popular for scene completion. We propose to fuse the scene reconstruction from multiple images and distill this knowledge into a more accurate single-view scene reconstruction.
• Score: 39.08243715525956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inferring scene geometry from images via Structure from Motion is a long-standing and fundamental problem in computer vision. While classical approaches and, more recently, depth map predictions only focus on the visible parts of a scene, the task of scene completion aims to reason about geometry even in occluded regions. With the popularity of neural radiance fields (NeRFs), implicit representations also became popular for scene completion by predicting so-called density fields. Unlike explicit approaches. e.g. voxel-based methods, density fields also allow for accurate depth prediction and novel-view synthesis via image-based rendering. In this work, we propose to fuse the scene reconstruction from multiple images and distill this knowledge into a more accurate single-view scene reconstruction. To this end, we propose Multi-View Behind the Scenes (MVBTS) to fuse density fields from multiple posed images, trained fully self-supervised only from image data. Using knowledge distillation, we use MVBTS to train a single-view scene completion network via direct supervision called KDBTS. It achieves state-of-the-art performance on occupancy prediction, especially in occluded regions.

Related papers

• DoubleTake: Geometry Guided Depth Estimation [17.464549832122714]
  Estimating depth from a sequence of posed RGB images is a fundamental computer vision task. We introduce a reconstruction which combines volume features with a hint of the prior geometry, rendered as a depth map from the current camera location. We demonstrate that our method can run at interactive speeds, state-of-the-art estimates of depth and 3D scene in both offline and incremental evaluation scenarios.
  arXiv  Detail & Related papers  (2024-06-26T14:29:05Z)
• Self-supervised Learning of Neural Implicit Feature Fields for Camera Pose Refinement [32.335953514942474]
  This paper proposes to jointly learn the scene representation along with a 3D dense feature field and a 2D feature extractor. We learn the underlying geometry of the scene with an implicit field through volumetric rendering and design our feature field to leverage intermediate geometric information encoded in the implicit field. Visual localization is then achieved by aligning the image-based features and the rendered volumetric features.
  arXiv  Detail & Related papers  (2024-06-12T17:51:53Z)
• Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning [119.99066522299309]
  KYN is a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work.
  arXiv  Detail & Related papers  (2024-04-04T17:59:59Z)
• 3D Reconstruction with Generalizable Neural Fields using Scene Priors [71.37871576124789]
  We introduce training generalizable Neural Fields incorporating scene Priors (NFPs) The NFP network maps any single-view RGB-D image into signed distance and radiance values. A complete scene can be reconstructed by merging individual frames in the volumetric space WITHOUT a fusion module.
  arXiv  Detail & Related papers  (2023-09-26T18:01:02Z)
• Behind the Scenes: Density Fields for Single View Reconstruction [63.40484647325238]
  Inferring meaningful geometric scene representation from a single image is a fundamental problem in computer vision. We propose to predict implicit density fields. A density field maps every location in the frustum of the input image to volumetric density. We show that our method is able to predict meaningful geometry for regions that are occluded in the input image.
  arXiv  Detail & Related papers  (2023-01-18T17:24:01Z)
• Deep scene-scale material estimation from multi-view indoor captures [9.232860902853048]
  We present a learning-based approach that automatically produces digital assets ready for physically-based rendering. Our method generates approximate material maps in a fraction of time compared to the closest previous solutions.
  arXiv  Detail & Related papers  (2022-11-15T10:58:28Z)
• IBRNet: Learning Multi-View Image-Based Rendering [67.15887251196894]
  We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering.
  arXiv  Detail & Related papers  (2021-02-25T18:56:21Z)
• Free View Synthesis [100.86844680362196]
  We present a method for novel view synthesis from input images that are freely distributed around a scene. Our method does not rely on a regular arrangement of input views, can synthesize images for free camera movement through the scene, and works for general scenes with unconstrained geometric layouts.
  arXiv  Detail & Related papers  (2020-08-12T18:16:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.