MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds

• URL: http://arxiv.org/abs/2412.06974v1
• Date: Mon, 09 Dec 2024 20:34:55 GMT
• Title: MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds
• Authors: Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, Zhicheng Yan,
• Abstract summary: We propose a fast single-stage feed-forward network MV-DUSt3R to handle more views, reduce errors, and improve inference time. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices.
• Score: 56.77548728485841
• License:
• Abstract: Recent sparse multi-view scene reconstruction advances like DUSt3R and MASt3R no longer require camera calibration and camera pose estimation. However, they only process a pair of views at a time to infer pixel-aligned pointmaps. When dealing with more than two views, a combinatorial number of error prone pairwise reconstructions are usually followed by an expensive global optimization, which often fails to rectify the pairwise reconstruction errors. To handle more views, reduce errors, and improve inference time, we propose the fast single-stage feed-forward network MV-DUSt3R. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices. To further enable novel view synthesis, we extend both by adding and jointly training Gaussian splatting heads. Experiments on multi-view stereo reconstruction, multi-view pose estimation, and novel view synthesis confirm that our methods improve significantly upon prior art. Code will be released.

Related papers

• Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass [68.78222900840132]
  We propose Fast 3D Reconstruction (Fast3R), a novel multi-view generalization to DUSt3R that achieves efficient and scalable 3D reconstruction by processing many views in parallel. Fast3R demonstrates state-of-the-art performance, with significant improvements in inference speed and reduced error accumulation.
  arXiv  Detail & Related papers  (2025-01-23T18:59:55Z)
• RCNet: Deep Recurrent Collaborative Network for Multi-View Low-Light Image Enhancement [19.751696790765635]
  We make the first attempt to investigate multi-view low-light image enhancement. We propose a deep multi-view enhancement framework based on the Recurrent Collaborative Network (RCNet) Experimental results demonstrate that our RCNet significantly outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-06T15:49:49Z)
• 2L3: Lifting Imperfect Generated 2D Images into Accurate 3D [16.66666619143761]
  Multi-view (MV) 3D reconstruction is a promising solution to fuse generated MV images into consistent 3D objects. However, the generated images usually suffer from inconsistent lighting, misaligned geometry, and sparse views, leading to poor reconstruction quality. We present a novel 3D reconstruction framework that leverages intrinsic decomposition guidance, transient-mono prior guidance, and view augmentation to cope with the three issues.
  arXiv  Detail & Related papers  (2024-01-29T02:30:31Z)
• DUSt3R: Geometric 3D Vision Made Easy [8.471330244002564]
  We introduce DUSt3R, a novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera.
  arXiv  Detail & Related papers  (2023-12-21T18:52:14Z)
• UpFusion: Novel View Diffusion from Unposed Sparse View Observations [66.36092764694502]
  UpFusion can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.
  arXiv  Detail & Related papers  (2023-12-11T18:59:55Z)
• Learning to Render Novel Views from Wide-Baseline Stereo Pairs [26.528667940013598]
  We introduce a method for novel view synthesis given only a single wide-baseline stereo image pair. Existing approaches to novel view synthesis from sparse observations fail due to recovering incorrect 3D geometry. We propose an efficient, image-space epipolar line sampling scheme to assemble image features for a target ray.
  arXiv  Detail & Related papers  (2023-04-17T17:40:52Z)
• VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion [68.68537312256144]
  VoRTX is an end-to-end volumetric 3D reconstruction network using transformers for wide-baseline, multi-view feature fusion. We train our model on ScanNet and show that it produces better reconstructions than state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-01T02:18:11Z)
• Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo [71.59494156155309]
  Existing approaches for multi-view 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views. We present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot.
  arXiv  Detail & Related papers  (2021-04-06T03:49:35Z)
• Monocular Depth Estimation with Self-supervised Instance Adaptation [138.0231868286184]
  In robotics applications, multiple views of a scene may or may not be available, depend-ing on the actions of the robot. We propose a new approach that extends any off-the-shelf self-supervised monocular depth reconstruction system to usemore than one image at test time.
  arXiv  Detail & Related papers  (2020-04-13T08:32:03Z)
• Learning to Correct 3D Reconstructions from Multiple Views [20.315829094519128]
  We render 2D views of an existing reconstruction and train a convolutional neural network that refines inverse-depth to match a higher-quality reconstruction. Since the views that we correct are rendered from the same reconstruction, they share the same geometry, so overlapping views complement each other. We propose a method for transforming features with dynamic filters generated by a multi-layer perceptron from the relative poses between views.
  arXiv  Detail & Related papers  (2020-01-22T16:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.