PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility

• URL: http://arxiv.org/abs/2108.08943v1
• Date: Thu, 19 Aug 2021 23:14:48 GMT
• Title: PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility
• Authors: Jae Yong Lee, Joseph DeGol, Chuhang Zou, Derek Hoiem
• Abstract summary: We propose an end-to-end trainable PatchMatch-based MVS approach that combines advantages of trainable costs and regularizations with pixelwise estimates. We evaluate our method on widely used MVS benchmarks, ETH3D and Tanks and Temples (TnT)
• Score: 23.427619869594437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent learning-based multi-view stereo (MVS) methods show excellent performance with dense cameras and small depth ranges. However, non-learning based approaches still outperform for scenes with large depth ranges and sparser wide-baseline views, in part due to their PatchMatch optimization over pixelwise estimates of depth, normals, and visibility. In this paper, we propose an end-to-end trainable PatchMatch-based MVS approach that combines advantages of trainable costs and regularizations with pixelwise estimates. To overcome the challenge of the non-differentiable PatchMatch optimization that involves iterative sampling and hard decisions, we use reinforcement learning to minimize expected photometric cost and maximize likelihood of ground truth depth and normals. We incorporate normal estimation by using dilated patch kernels, and propose a recurrent cost regularization that applies beyond frontal plane-sweep algorithms to our pixelwise depth/normal estimates. We evaluate our method on widely used MVS benchmarks, ETH3D and Tanks and Temples (TnT), and compare to other state of the art learning based MVS models. On ETH3D, our method outperforms other recent learning-based approaches and performs comparably on advanced TnT.

Related papers

• MP-MVS: Multi-Scale Windows PatchMatch and Planar Prior Multi-View Stereo [7.130834755320434]
  We propose a resilient and effective multi-view stereo approach (MP-MVS) We design a multi-scale windows PatchMatch (mPM) to obtain reliable depth of untextured areas. In contrast with other multi-scale approaches, which is faster and can be easily extended to PatchMatch-based MVS approaches.
  arXiv  Detail & Related papers  (2023-09-23T07:30:42Z)
• Single Image Depth Prediction Made Better: A Multivariate Gaussian Take [163.14849753700682]
  We introduce an approach that performs continuous modeling of per-pixel depth. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.
  arXiv  Detail & Related papers  (2023-03-31T16:01:03Z)
• Deep PatchMatch MVS with Learned Patch Coplanarity, Geometric Consistency and Adaptive Pixel Sampling [19.412014102866507]
  We build on learning-based approaches to improve photometric scores by learning patch coplanarity and encourage geometric consistency. We propose an adaptive pixel sampling strategy for candidate propagation that reduces memory to enable training on larger resolution with more views and a larger encoder.
  arXiv  Detail & Related papers  (2022-10-14T07:29:03Z)
• Revisiting PatchMatch Multi-View Stereo for Urban 3D Reconstruction [1.1011268090482573]
  A complete pipeline for image-based 3D reconstruction of urban scenarios is proposed, based on PatchMatch Multi-View Stereo (MVS) The proposed approach is carefully evaluated against both classical MVS algorithms and monocular depth networks on the KITTI dataset.
  arXiv  Detail & Related papers  (2022-07-18T08:45:54Z)
• PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction [2.9896482273918434]
  This paper proposes robust loss functions leveraging constraints beneath multi-view images to alleviate matching ambiguity. Our strategy can be implemented with arbitrary depth estimation frameworks and can be trained with arbitrary large-scale MVS datasets. Our method reaches the performance of the state-of-the-art methods on popular benchmarks, like DTU, Tanks and Temples and ETH3D.
  arXiv  Detail & Related papers  (2022-03-04T07:05:23Z)
• IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo [71.84742490020611]
  IterMVS is a new data-driven method for high-resolution multi-view stereo. We propose a novel GRU-based estimator that encodes pixel-wise probability distributions of depth in its hidden state. We verify the efficiency and effectiveness of our method on DTU, Tanks&Temples and ETH3D.
  arXiv  Detail & Related papers  (2021-12-09T18:58:02Z)
• Generalized Binary Search Network for Highly-Efficient Multi-View Stereo [10.367295443948487]
  Multi-view Stereo (MVS) with known camera parameters is essentially a 1D search problem within a valid depth range. Recent deep learning-based MVS methods typically densely sample depth hypotheses in the depth range. We propose a novel method for highly efficient MVS that remarkably decreases the memory footprint.
  arXiv  Detail & Related papers  (2021-12-04T13:57:18Z)
• An Adaptive Framework for Learning Unsupervised Depth Completion [59.17364202590475]
  We present a method to infer a dense depth map from a color image and associated sparse depth measurements. We show that regularization and co-visibility are related via the fitness of the model to data and can be unified into a single framework.
  arXiv  Detail & Related papers  (2021-06-06T02:27:55Z)
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction [87.08227378010874]
  We show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces a simple geometric constraint, we significantly improve the accuracy and robustness of monocular depth estimation. We show state-of-the-art results of learning metric depth on NYU Depth-V2 and KITTI.
  arXiv  Detail & Related papers  (2021-03-07T00:08:21Z)
• PatchmatchNet: Learned Multi-View Patchmatch Stereo [70.14789588576438]
  We present PatchmatchNet, a novel and learnable cascade formulation of Patchmatch for high-resolution multi-view stereo. With high speed and low memory requirement, PatchmatchNet can process higher resolution imagery and is more suited to run on resource limited devices than competitors that employ 3D cost volume regularization.
  arXiv  Detail & Related papers  (2020-12-02T18:59:02Z)
• Deep Shells: Unsupervised Shape Correspondence with Optimal Transport [52.646396621449]
  We propose a novel unsupervised learning approach to 3D shape correspondence. We show that the proposed method significantly improves over the state-of-the-art on multiple datasets.
  arXiv  Detail & Related papers  (2020-10-28T22:24:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.