RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

• URL: http://arxiv.org/abs/2203.03949v1
• Date: Tue, 8 Mar 2022 09:24:05 GMT
• Title: RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering
• Authors: Di Chang, Alja\v{z} Bo\v{z}i\v{c}, Tong Zhang, Qingsong Yan, Yingcong Chen, Sabine S\"usstrunk, Matthias Nie{\ss}ner
• Abstract summary: We propose a novel approach with neural rendering (RC-MVSNet) to solve ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface. We also introduce a reference view loss to generate consistent supervision, even for non-Lambertian surfaces.
• Score: 16.679446000660654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Finding accurate correspondences among different views is the Achilles' heel of unsupervised Multi-View Stereo (MVS). Existing methods are built upon the assumption that corresponding pixels share similar photometric features. However, multi-view images in real scenarios observe non-Lambertian surfaces and experience occlusions. In this work, we propose a novel approach with neural rendering (RC-MVSNet) to solve such ambiguity issues of correspondences among views. Specifically, we impose a depth rendering consistency loss to constrain the geometry features close to the object surface to alleviate occlusions. Concurrently, we introduce a reference view synthesis loss to generate consistent supervision, even for non-Lambertian surfaces. Extensive experiments on DTU and Tanks\&Temples benchmarks demonstrate that our RC-MVSNet approach achieves state-of-the-art performance over unsupervised MVS frameworks and competitive performance to many supervised methods.The trained models and code will be released at https://github.com/Boese0601/RC-MVSNet.

Related papers

• A Global Depth-Range-Free Multi-View Stereo Transformer Network with Pose Embedding [76.44979557843367]
  We propose a novel multi-view stereo (MVS) framework that gets rid of the depth range prior. We introduce a Multi-view Disparity Attention (MDA) module to aggregate long-range context information. We explicitly estimate the quality of the current pixel corresponding to sampled points on the epipolar line of the source image.
  arXiv  Detail & Related papers  (2024-11-04T08:50:16Z)
• GenS: Generalizable Neural Surface Reconstruction from Multi-View Images [20.184657468900852]
  GenS is an end-to-end generalizable neural surface reconstruction model. Our representation is more powerful, which can recover high-frequency details while maintaining global smoothness. Experiments on popular benchmarks show that our model can generalize well to new scenes.
  arXiv  Detail & Related papers  (2024-06-04T17:13:10Z)
• S-VolSDF: Sparse Multi-View Stereo Regularization of Neural Implicit Surfaces [75.30792581941789]
  Neural rendering of implicit surfaces performs well in 3D vision applications. When only sparse input images are available, output quality drops significantly due to the shape-radiance ambiguity problem. We propose to regularize neural rendering optimization with an MVS solution.
  arXiv  Detail & Related papers  (2023-03-30T21:10:58Z)
• VS-Net: Multiscale Spatiotemporal Features for Lightweight Video Salient Document Detection [0.2578242050187029]
  We propose VS-Net, which captures multi-scaletemporal information with the help of dilated depth-wise separable convolution and Approximation Rank Pooling. Our model generates saliency maps considering both the background and foreground, making it perform better in challenging scenarios. The immense experiments regulated on the benchmark MIDV-500 dataset show that the VS-Net model outperforms state-of-the-art approaches in both time and robustness measures.
  arXiv  Detail & Related papers  (2023-01-11T13:07:31Z)
• Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing [71.19528222206088]
  We propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. Our method achieves the new state-of-the-art performance on the Helen, CelebA-HQ, and LapaMask datasets.
  arXiv  Detail & Related papers  (2022-03-28T02:12:30Z)
• PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction [2.9896482273918434]
  This paper proposes robust loss functions leveraging constraints beneath multi-view images to alleviate matching ambiguity. Our strategy can be implemented with arbitrary depth estimation frameworks and can be trained with arbitrary large-scale MVS datasets. Our method reaches the performance of the state-of-the-art methods on popular benchmarks, like DTU, Tanks and Temples and ETH3D.
  arXiv  Detail & Related papers  (2022-03-04T07:05:23Z)
• TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo [55.30992853477754]
  We present TANDEM, a real-time monocular tracking and dense framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of alignments. TANDEM shows state-of-the-art real-time 3D reconstruction performance.
  arXiv  Detail & Related papers  (2021-11-14T19:01:02Z)
• Self-Supervised Visibility Learning for Novel View Synthesis [79.53158728483375]
  Conventional rendering methods estimate scene geometry and synthesize novel views in two separate steps. We propose an end-to-end NVS framework to eliminate the error propagation issue. Our network is trained in an end-to-end self-supervised fashion, thus significantly alleviating error accumulation in view synthesis.
  arXiv  Detail & Related papers  (2021-03-29T08:11:25Z)
• SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis [52.35697180864202]
  Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. We investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. We present SurfaceNet+, a volumetric method to handle the 'incompleteness' and the 'inaccuracy' problems induced by a very sparse MVS setup.
  arXiv  Detail & Related papers  (2020-05-26T13:13:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.