Learning to Correct 3D Reconstructions from Multiple Views

• URL: http://arxiv.org/abs/2001.08098v1
• Date: Wed, 22 Jan 2020 16:02:23 GMT
• Title: Learning to Correct 3D Reconstructions from Multiple Views
• Authors: \c{S}tefan S\u{a}ftescu and Paul Newman
• Abstract summary: We render 2D views of an existing reconstruction and train a convolutional neural network that refines inverse-depth to match a higher-quality reconstruction. Since the views that we correct are rendered from the same reconstruction, they share the same geometry, so overlapping views complement each other. We propose a method for transforming features with dynamic filters generated by a multi-layer perceptron from the relative poses between views.
• Score: 20.315829094519128
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper is about reducing the cost of building good large-scale 3D reconstructions post-hoc. We render 2D views of an existing reconstruction and train a convolutional neural network (CNN) that refines inverse-depth to match a higher-quality reconstruction. Since the views that we correct are rendered from the same reconstruction, they share the same geometry, so overlapping views complement each other. We take advantage of that in two ways. Firstly, we impose a loss during training which guides predictions on neighbouring views to have the same geometry and has been shown to improve performance. Secondly, in contrast to previous work, which corrects each view independently, we also make predictions on sets of neighbouring views jointly. This is achieved by warping feature maps between views and thus bypassing memory-intensive 3D computation. We make the observation that features in the feature maps are viewpoint-dependent, and propose a method for transforming features with dynamic filters generated by a multi-layer perceptron from the relative poses between views. In our experiments we show that this last step is necessary for successfully fusing feature maps between views.

Related papers

• MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds [56.77548728485841]
  We propose a fast single-stage feed-forward network MV-DUSt3R to handle more views, reduce errors, and improve inference time. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices.
  arXiv  Detail & Related papers  (2024-12-09T20:34:55Z)
• DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction [67.13370009386635]
  We introduce the Dual Point Maps (DualPM), where a pair of point maps is extracted from the same image, one associating pixels to their 3D locations on the object, and the other to a canonical version of the object at rest pose. We show that 3D reconstruction and 3D pose estimation reduce to the prediction of the DualPMs.
  arXiv  Detail & Related papers  (2024-12-05T18:59:48Z)
• Neural Semantic Surface Maps [52.61017226479506]
  We present an automated technique for computing a map between two genus-zero shapes, which matches semantically corresponding regions to one another. Our approach can generate semantic surface-to-surface maps, eliminating manual annotations or any 3D training data requirement.
  arXiv  Detail & Related papers  (2023-09-09T16:21:56Z)
• RelPose++: Recovering 6D Poses from Sparse-view Observations [66.6922660401558]
  We address the task of estimating 6D camera poses from sparse-view image sets (2-8 images) We build on the recent RelPose framework which learns a network that infers distributions over relative rotations over image pairs. Our final system results in large improvements in 6D pose prediction over prior art on both seen and unseen object categories.
  arXiv  Detail & Related papers  (2023-05-08T17:59:58Z)
• Single-view 3D Mesh Reconstruction for Seen and Unseen Categories [69.29406107513621]
  Single-view 3D Mesh Reconstruction is a fundamental computer vision task that aims at recovering 3D shapes from single-view RGB images. This paper tackles Single-view 3D Mesh Reconstruction, to study the model generalization on unseen categories. We propose an end-to-end two-stage network, GenMesh, to break the category boundaries in reconstruction.
  arXiv  Detail & Related papers  (2022-08-04T14:13:35Z)
• Fine Detailed Texture Learning for 3D Meshes with Generative Models [33.42114674602613]
  This paper presents a method to reconstruct high-quality textured 3D models from both multi-view and single-view images. In the first stage, we focus on learning accurate geometry, whereas in the second stage, we focus on learning the texture with a generative adversarial network. We demonstrate that our method achieves superior 3D textured models compared to the previous works.
  arXiv  Detail & Related papers  (2022-03-17T14:50:52Z)
• Improving neural implicit surfaces geometry with patch warping [12.106051690920266]
  We argue that this comes from the difficulty to learn and render high frequency textures with neural networks. We propose to add to the standard neural rendering optimization a direct photo-consistency term across the different views. We evaluate our approach, dubbed NeuralWarp, on the standard DTU and EPFL benchmarks and show it outperforms state of the art unsupervised implicit surfaces reconstructions by over 20% on both datasets.
  arXiv  Detail & Related papers  (2021-12-17T17:43:50Z)
• A Divide et Impera Approach for 3D Shape Reconstruction from Multiple Views [49.03830902235915]
  Estimating the 3D shape of an object from a single or multiple images has gained popularity thanks to the recent breakthroughs powered by deep learning. This paper proposes to rely on viewpoint variant reconstructions by merging the visible information from the given views. To validate the proposed method, we perform a comprehensive evaluation on the ShapeNet reference benchmark in terms of relative pose estimation and 3D shape reconstruction.
  arXiv  Detail & Related papers  (2020-11-17T09:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.