UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Sets

• URL: http://arxiv.org/abs/2403.05086v3
• Date: Fri, 17 May 2024 13:17:50 GMT
• Title: UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Sets
• Authors: Youngju Na, Woo Jae Kim, Kyu Beom Han, Suhyeon Ha, Sung-eui Yoon,
• Abstract summary: We introduce and validate a view-combination score to indicate the effectiveness of the input view combination. To achieve this, we apply cross-view matching transformers to model interactions between source images and build correlation frustums. Our proposed framework significantly outperforms previous methods in terms of view-combination generalizability.
• Score: 20.767590006724117
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generalizable neural implicit surface reconstruction aims to obtain an accurate underlying geometry given a limited number of multi-view images from unseen scenes. However, existing methods select only informative and relevant views using predefined scores for training and testing phases. This constraint renders the model impractical in real-world scenarios, where the availability of favorable combinations cannot always be ensured. We introduce and validate a view-combination score to indicate the effectiveness of the input view combination. We observe that previous methods output degenerate solutions under arbitrary and unfavorable sets. Building upon this finding, we propose UFORecon, a robust view-combination generalizable surface reconstruction framework. To achieve this, we apply cross-view matching transformers to model interactions between source images and build correlation frustums to capture global correlations. Additionally, we explicitly encode pairwise feature similarities as view-consistent priors. Our proposed framework significantly outperforms previous methods in terms of view-combination generalizability and also in the conventional generalizable protocol trained with favorable view-combinations. The code is available at https://github.com/Youngju-Na/UFORecon.

Related papers

• Joint Linked Component Analysis for Multiview Data [6.588932144201398]
  We formulate a matrix decomposition model where a joint structure and an individual structure are present in each data view. An objective function with a novel penalty term is then proposed to achieve simultaneous estimation and rank selection.
  arXiv  Detail & Related papers  (2024-06-17T17:25:23Z)
• Multi-Scene Generalized Trajectory Global Graph Solver with Composite Nodes for Multiple Object Tracking [61.69892497726235]
  Composite Node Message Passing Network (CoNo-Link) is a framework for modeling ultra-long frames information for association. In addition to the previous method of treating objects as nodes, the network innovatively treats object trajectories as nodes for information interaction. Our model can learn better predictions on longer-time scales by adding composite nodes.
  arXiv  Detail & Related papers  (2023-12-14T14:00:30Z)
• UpFusion: Novel View Diffusion from Unposed Sparse View Observations [66.36092764694502]
  UpFusion can perform novel view synthesis and infer 3D representations for an object given a sparse set of reference images. We show that this mechanism allows generating high-fidelity novel views while improving the synthesis quality given additional (unposed) images.
  arXiv  Detail & Related papers  (2023-12-11T18:59:55Z)
• Handbook on Leveraging Lines for Two-View Relative Pose Estimation [82.72686460985297]
  We propose an approach for estimating the relative pose between image pairs by jointly exploiting points, lines, and their coincidences in a hybrid manner. Our hybrid framework combines the advantages of all configurations, enabling robust and accurate estimation in challenging environments.
  arXiv  Detail & Related papers  (2023-09-27T21:43:04Z)
• Rotation-Constrained Cross-View Feature Fusion for Multi-View Appearance-based Gaze Estimation [16.43119580796718]
  This work proposes a generalizable multi-view gaze estimation task and a cross-view feature fusion method to address this issue. In addition to paired images, our method takes the relative rotation matrix between two cameras as additional input. The proposed network learns to extract rotatable feature representation by using relative rotation as a constraint.
  arXiv  Detail & Related papers  (2023-05-22T04:29:34Z)
• Explicit Correspondence Matching for Generalizable Neural Radiance Fields [49.49773108695526]
  We present a new NeRF method that is able to generalize to new unseen scenarios and perform novel view synthesis with as few as two source views. The explicit correspondence matching is quantified with the cosine similarity between image features sampled at the 2D projections of a 3D point on different views. Our method achieves state-of-the-art results on different evaluation settings, with the experiments showing a strong correlation between our learned cosine feature similarity and volume density.
  arXiv  Detail & Related papers  (2023-04-24T17:46:01Z)
• RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild [73.1276968007689]
  We describe a data-driven method for inferring the camera viewpoints given multiple images of an arbitrary object. We show that our approach outperforms state-of-the-art SfM and SLAM methods given sparse images on both seen and unseen categories.
  arXiv  Detail & Related papers  (2022-08-11T17:59:59Z)
• AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild [77.43884383743872]
  We present AdaFuse, an adaptive multiview fusion method to enhance the features in occluded views. We extensively evaluate the approach on three public datasets including Human3.6M, Total Capture and CMU Panoptic. We also create a large scale synthetic dataset Occlusion-Person, which allows us to perform numerical evaluation on the occluded joints.
  arXiv  Detail & Related papers  (2020-10-26T03:19:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.