Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs

• URL: http://arxiv.org/abs/2312.07246v2
• Date: Mon, 8 Apr 2024 07:07:02 GMT
• Title: Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairs
• Authors: Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, Chong Luo,
• Abstract summary: This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks.
• Score: 57.492124844326206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work delves into the task of pose-free novel view synthesis from stereo pairs, a challenging and pioneering task in 3D vision. Our innovative framework, unlike any before, seamlessly integrates 2D correspondence matching, camera pose estimation, and NeRF rendering, fostering a synergistic enhancement of these tasks. We achieve this through designing an architecture that utilizes a shared representation, which serves as a foundation for enhanced 3D geometry understanding. Capitalizing on the inherent interplay between the tasks, our unified framework is trained end-to-end with the proposed training strategy to improve overall model accuracy. Through extensive evaluations across diverse indoor and outdoor scenes from two real-world datasets, we demonstrate that our approach achieves substantial improvement over previous methodologies, especially in scenarios characterized by extreme viewpoint changes and the absence of accurate camera poses.

Related papers

• RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors [13.883695200241524]
  RI3D is a novel approach that harnesses the power of diffusion models to reconstruct high-quality novel views given a sparse set of input images. Our key contribution is separating the view synthesis process into two tasks of reconstructing visible regions and hallucinating missing regions. We produce results with detailed textures in both visible and missing regions that outperform state-of-the-art approaches on a diverse set of scenes.
  arXiv  Detail & Related papers  (2025-03-13T20:16:58Z)
• GarmentCrafter: Progressive Novel View Synthesis for Single-View 3D Garment Reconstruction and Editing [85.67881477813592]
  GarmentCrafter is a new approach that enables non-professional users to create and modify 3D garments from a single-view image. Our method achieves superior visual fidelity and inter-view coherence compared to state-of-the-art single-view 3D garment reconstruction methods.
  arXiv  Detail & Related papers  (2025-03-11T17:56:03Z)
• Evolving High-Quality Rendering and Reconstruction in a Unified Framework with Contribution-Adaptive Regularization [27.509109317973817]
  3D Gaussian Splatting (3DGS) has garnered significant attention for its high-quality rendering and fast inference speed. Previous methods primarily focus on geometry regularization, with common approaches including primitive-based and dual-model frameworks. We propose CarGS, a unified model leveraging-adaptive regularization to achieve simultaneous, high-quality surface reconstruction.
  arXiv  Detail & Related papers  (2025-03-02T12:51:38Z)
• Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis [25.898616784744377]
  Given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D. We propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them.
  arXiv  Detail & Related papers  (2024-12-04T18:59:24Z)
• Integrating View Conditions for Image Synthesis [14.738884513493227]
  This paper introduces a pioneering framework that integrates viewpoint information to enhance the control of image editing tasks. We distill three essential criteria -- consistency, controllability, and harmony -- that should be met for an image editing method.
  arXiv  Detail & Related papers  (2023-10-24T16:55:07Z)
• Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis [37.98068169673019]
  Implicit neural representations have shown powerful capacity in modeling real-world 3D scenes, offering superior performance in novel view synthesis. We propose a unified Neural Radiance Field (NeRF) framework to effectively perform joint scene decomposition and composition.
  arXiv  Detail & Related papers  (2023-08-05T10:42:05Z)
• Towards Scalable Multi-View Reconstruction of Geometry and Materials [27.660389147094715]
  We propose a novel method for joint recovery of camera pose, object geometry and spatially-varying Bidirectional Reflectance Distribution Function (svBRDF) of 3D scenes. The input are high-resolution RGBD images captured by a mobile, hand-held capture system with point lights for active illumination.
  arXiv  Detail & Related papers  (2023-06-06T15:07:39Z)
• Unifying Flow, Stereo and Depth Estimation [121.54066319299261]
  We present a unified formulation and model for three motion and 3D perception tasks. We formulate all three tasks as a unified dense correspondence matching problem. Our model naturally enables cross-task transfer since the model architecture and parameters are shared across tasks.
  arXiv  Detail & Related papers  (2022-11-10T18:59:54Z)
• Robust Single Image Dehazing Based on Consistent and Contrast-Assisted Reconstruction [95.5735805072852]
  We propose a novel density-variational learning framework to improve the robustness of the image dehzing model. Specifically, the dehazing network is optimized under the consistency-regularized framework. Our method significantly surpasses the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T08:11:04Z)
• Self-Supervised Multi-View Synchronization Learning for 3D Pose Estimation [39.334995719523]
  Current methods cast monocular 3D human pose estimation as a learning problem by training neural networks on large data sets of images and corresponding skeleton poses. We propose an approach that can exploit small annotated data sets by fine-tuning networks pre-trained via self-supervised learning on (large) unlabeled data sets. We demonstrate the effectiveness of the synchronization task on the Human3.6M data set and achieve state-of-the-art results in 3D human pose estimation.
  arXiv  Detail & Related papers  (2020-10-13T08:01:24Z)
• Towards Realistic 3D Embedding via View Alignment [53.89445873577063]
  This paper presents an innovative View Alignment GAN (VA-GAN) that composes new images by embedding 3D models into 2D background images realistically and automatically. VA-GAN consists of a texture generator and a differential discriminator that are inter-connected and end-to-end trainable.
  arXiv  Detail & Related papers  (2020-07-14T14:45:00Z)
• PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction [67.08350202974434]
  We propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. We show that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.
  arXiv  Detail & Related papers  (2020-07-08T02:26:19Z)
• Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild [101.70320427145388]
  We propose a weakly-supervised approach that does not require 3D annotations and learns to estimate 3D poses from unlabeled multi-view data. We evaluate our proposed approach on two large scale datasets.
  arXiv  Detail & Related papers  (2020-03-17T08:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.