Learning-based Multi-View Stereo: A Survey

• URL: http://arxiv.org/abs/2408.15235v1
• Date: Tue, 27 Aug 2024 17:53:18 GMT
• Title: Learning-based Multi-View Stereo: A Survey
• Authors: Fangjinhua Wang, Qingtian Zhu, Di Chang, Quankai Gao, Junlin Han, Tong Zhang, Richard Hartley, Marc Pollefeys,
• Abstract summary: Multi-View Stereo (MVS) algorithms synthesize a comprehensive 3D representation, enabling precise reconstruction in complex environments. With the success of deep learning, many learning-based MVS methods have been proposed, achieving impressive performance against traditional methods.
• Score: 55.3096230732874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D reconstruction aims to recover the dense 3D structure of a scene. It plays an essential role in various applications such as Augmented/Virtual Reality (AR/VR), autonomous driving and robotics. Leveraging multiple views of a scene captured from different viewpoints, Multi-View Stereo (MVS) algorithms synthesize a comprehensive 3D representation, enabling precise reconstruction in complex environments. Due to its efficiency and effectiveness, MVS has become a pivotal method for image-based 3D reconstruction. Recently, with the success of deep learning, many learning-based MVS methods have been proposed, achieving impressive performance against traditional methods. We categorize these learning-based methods as: depth map-based, voxel-based, NeRF-based, 3D Gaussian Splatting-based, and large feed-forward methods. Among these, we focus significantly on depth map-based methods, which are the main family of MVS due to their conciseness, flexibility and scalability. In this survey, we provide a comprehensive review of the literature at the time of this writing. We investigate these learning-based methods, summarize their performances on popular benchmarks, and discuss promising future research directions in this area.

Related papers

• Deep Models for Multi-View 3D Object Recognition: A Review [16.500711021549947]
  Multi-view 3D representations for object recognition has thus far demonstrated the most promising results for achieving state-of-the-art performance. This review paper comprehensively covers recent progress in multi-view 3D object recognition methods for 3D classification and retrieval tasks.
  arXiv  Detail & Related papers  (2024-04-23T16:54:31Z)
• Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting [32.66151412557986]
  We present a weak-to-strong eliciting framework aimed at enhancing surround refinement while maintaining robust monocular perception. Our framework employs weakly tuned experts trained on distinct subsets, and each is inherently biased toward specific camera configurations and scenarios. For MC3D-Det joint training, the elaborate dataset merge strategy is designed to solve the problem of inconsistent camera numbers and camera parameters.
  arXiv  Detail & Related papers  (2024-04-10T03:11:10Z)
• RadOcc: Learning Cross-Modality Occupancy Knowledge through Rendering Assisted Distillation [50.35403070279804]
  3D occupancy prediction is an emerging task that aims to estimate the occupancy states and semantics of 3D scenes using multi-view images. We propose RadOcc, a Rendering assisted distillation paradigm for 3D Occupancy prediction.
  arXiv  Detail & Related papers  (2023-12-19T03:39:56Z)
• One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception [59.37727312705997]
  This paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD. For the SSC task, our work stands out as the first to surpass LiDAR-based methods on the Semantic KITTI dataset.
  arXiv  Detail & Related papers  (2023-06-22T05:55:53Z)
• Cross-Dimensional Refined Learning for Real-Time 3D Visual Perception from Monocular Video [2.2299983745857896]
  We present a novel real-time capable learning method that jointly perceives a 3D scene's geometry structure and semantic labels. We propose an end-to-end cross-dimensional refinement neural network (CDRNet) to extract both 3D mesh and 3D semantic labeling in real time.
  arXiv  Detail & Related papers  (2023-03-16T11:53:29Z)
• State of the Art in Dense Monocular Non-Rigid 3D Reconstruction [100.9586977875698]
  3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics. This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views.
  arXiv  Detail & Related papers  (2022-10-27T17:59:53Z)
• End-to-End Multi-View Structure-from-Motion with Hypercorrelation Volumes [7.99536002595393]
  Deep learning techniques have been proposed to tackle this problem. We improve on the state-of-the-art two-view structure-from-motion(SfM) approach. We extend it to the general multi-view case and evaluate it on the complex benchmark dataset DTU.
  arXiv  Detail & Related papers  (2022-09-14T20:58:44Z)
• 3DVNet: Multi-View Depth Prediction and Volumetric Refinement [68.68537312256144]
  3DVNet is a novel multi-view stereo (MVS) depth-prediction method. Our key idea is the use of a 3D scene-modeling network that iteratively updates a set of coarse depth predictions. We show that our method exceeds state-of-the-art accuracy in both depth prediction and 3D reconstruction metrics.
  arXiv  Detail & Related papers  (2021-12-01T00:52:42Z)
• Deep Learning for Multi-View Stereo via Plane Sweep: A Survey [0.0]
  3D reconstruction has lately attracted increasing attention due to its wide application in many areas, such as autonomous driving, robotics and virtual reality. As a dominant technique in artificial intelligence, deep learning has been successfully adopted to solve various computer vision problems. This paper presents a review of recent progress in deep learning methods for Multi-view Stereo (MVS), which is considered as a crucial task of image-based 3D reconstruction.
  arXiv  Detail & Related papers  (2021-06-18T14:10:44Z)
• Video Super Resolution Based on Deep Learning: A Comprehensive Survey [87.30395002197344]
  We comprehensively investigate 33 state-of-the-art video super-resolution (VSR) methods based on deep learning. We propose a taxonomy and classify the methods into six sub-categories according to the ways of utilizing inter-frame information. We summarize and compare the performance of the representative VSR method on some benchmark datasets.
  arXiv  Detail & Related papers  (2020-07-25T13:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.