TSAR-MVS: Textureless-aware Segmentation and Correlative Refinement Guided Multi-View Stereo

• URL: http://arxiv.org/abs/2308.09990v4
• Date: Fri, 30 Aug 2024 07:32:50 GMT
• Title: TSAR-MVS: Textureless-aware Segmentation and Correlative Refinement Guided Multi-View Stereo
• Authors: Zhenlong Yuan, Jiakai Cao, Zhaoqi Wang, Zhaoxin Li,
• Abstract summary: We propose a Textureless-aware And Correlative Refinement guided Multi-View Stereo (TSAR-MVS) method. It effectively tackles challenges posed by textureless areas in 3D reconstruction through filtering, refinement and segmentation. Experiments on ETH3D, Tanks & Temples and Strecha datasets demonstrate the superior performance and strong capability of our proposed method.
• Score: 3.6728185343140685
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The reconstruction of textureless areas has long been a challenging problem in MVS due to lack of reliable pixel correspondences between images. In this paper, we propose the Textureless-aware Segmentation And Correlative Refinement guided Multi-View Stereo (TSAR-MVS), a novel method that effectively tackles challenges posed by textureless areas in 3D reconstruction through filtering, refinement and segmentation. First, we implement the joint hypothesis filtering, a technique that merges a confidence estimator with a disparity discontinuity detector to eliminate incorrect depth estimations. Second, to spread the pixels with confident depth, we introduce an iterative correlation refinement strategy that leverages RANSAC to generate 3D planes based on superpixels, succeeded by a weighted median filter for broadening the influence of accurately determined pixels. Finally, we present a textureless-aware segmentation method that leverages edge detection and line detection for accurately identify large textureless regions for further depth completion. Experiments on ETH3D, Tanks & Temples and Strecha datasets demonstrate the superior performance and strong generalization capability of our proposed method.

Related papers

• Mode-GS: Monocular Depth Guided Anchored 3D Gaussian Splatting for Robust Ground-View Scene Rendering [47.879695094904015]
  We present a novelview rendering algorithm, Mode-GS, for ground-robot trajectory datasets. Our approach is based on using anchored Gaussian splats, which are designed to overcome the limitations of existing 3D Gaussian splatting algorithms. Our method results in improved rendering performance, based on PSNR, SSIM, and LPIPS metrics, in ground scenes with free trajectory patterns.
  arXiv  Detail & Related papers  (2024-10-06T23:01:57Z)
• GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision [49.839374549646884]
  This paper presents GEOcc, a Geometric-Enhanced Occupancy network tailored for vision-only surround-view perception. Our approach achieves State-Of-The-Art performance on the Occ3D-nuScenes dataset with the least image resolution needed and the most weightless image backbone.
  arXiv  Detail & Related papers  (2024-05-17T07:31:20Z)
• Depth-aware Volume Attention for Texture-less Stereo Matching [67.46404479356896]
  We propose a lightweight volume refinement scheme to tackle the texture deterioration in practical outdoor scenarios. We introduce a depth volume supervised by the ground-truth depth map, capturing the relative hierarchy of image texture. Local fine structure and context are emphasized to mitigate ambiguity and redundancy during volume aggregation.
  arXiv  Detail & Related papers  (2024-02-14T04:07:44Z)
• SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization [6.886220026399106]
  We introduce Multi-View Stereo (SD-MVS) to tackle challenges in 3D reconstruction of textureless areas. We are the first to adopt the Segment Anything Model (SAM) to distinguish semantic instances in scenes. We propose a unique refinement strategy that combines spherical coordinates and gradient descent on normals and pixelwise search interval on depths.
  arXiv  Detail & Related papers  (2024-01-12T05:25:57Z)
• Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity [17.98608948955211]
  Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume. We propose a disparity-based MVS method based on the epipolar disparity flow (E-flow), called DispMVS. We show that DispMVS is not sensitive to the depth range and achieves state-of-the-art results with lower GPU memory.
  arXiv  Detail & Related papers  (2022-11-30T11:05:02Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• On Robust Cross-View Consistency in Self-Supervised Monocular Depth Estimation [56.97699793236174]
  We study two kinds of robust cross-view consistency in this paper. We exploit the temporal coherence in both depth feature space and 3D voxel space for self-supervised monocular depth estimation. Experimental results on several outdoor benchmarks show that our method outperforms current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2022-09-19T03:46:13Z)
• PatchMVSNet: Patch-wise Unsupervised Multi-View Stereo for Weakly-Textured Surface Reconstruction [2.9896482273918434]
  This paper proposes robust loss functions leveraging constraints beneath multi-view images to alleviate matching ambiguity. Our strategy can be implemented with arbitrary depth estimation frameworks and can be trained with arbitrary large-scale MVS datasets. Our method reaches the performance of the state-of-the-art methods on popular benchmarks, like DTU, Tanks and Temples and ETH3D.
  arXiv  Detail & Related papers  (2022-03-04T07:05:23Z)
• IB-MVS: An Iterative Algorithm for Deep Multi-View Stereo based on Binary Decisions [0.0]
  We present a novel deep-learning-based method for Multi-View Stereo. Our method estimates high resolution and highly precise depth maps iteratively, by traversing the continuous space of feasible depth values at each pixel in a binary decision fashion. We compare our method with state-of-the-art Multi-View Stereo methods on the DTU, Tanks and Temples and the challenging ETH3D benchmarks and show competitive results.
  arXiv  Detail & Related papers  (2021-11-29T10:04:24Z)
• Shape Prior Non-Uniform Sampling Guided Real-time Stereo 3D Object Detection [59.765645791588454]
  Recently introduced RTS3D builds an efficient 4D Feature-Consistency Embedding space for the intermediate representation of object without depth supervision. We propose a shape prior non-uniform sampling strategy that performs dense sampling in outer region and sparse sampling in inner region. Our proposed method has 2.57% improvement on AP3d almost without extra network parameters.
  arXiv  Detail & Related papers  (2021-06-18T09:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.