Enhanced Stable View Synthesis

• URL: http://arxiv.org/abs/2303.17094v1
• Date: Thu, 30 Mar 2023 01:53:14 GMT
• Title: Enhanced Stable View Synthesis
• Authors: Nishant Jain, Suryansh Kumar, Luc Van Gool
• Abstract summary: We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging.
• Score: 86.69338893753886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging, leading to inferior results using the state-of-the-art stable view synthesis (SVS) method. SVS and related methods fail for outdoor scenes primarily due to (i) over-relying on the multiview stereo (MVS) for geometric scaffold recovery and (ii) assuming COLMAP computed camera poses as the best possible estimates, despite it being well-studied that MVS 3D reconstruction accuracy is limited to scene disparity and camera-pose accuracy is sensitive to key-point correspondence selection. This work proposes a principled way to enhance novel view synthesis solutions drawing inspiration from the basics of multiple view geometry. By leveraging the complementary behavior of MVS and monocular depth, we arrive at a better scene depth per view for nearby and far points, respectively. Moreover, our approach jointly refines camera poses with image-based rendering via multiple rotation averaging graph optimization. The recovered scene depth and the camera-pose help better view-dependent on-surface feature aggregation of the entire scene. Extensive evaluation of our approach on the popular benchmark dataset, such as Tanks and Temples, shows substantial improvement in view synthesis results compared to the prior art. For instance, our method shows 1.5 dB of PSNR improvement on the Tank and Temples. Similar statistics are observed when tested on other benchmark datasets such as FVS, Mip-NeRF 360, and DTU.

Related papers

• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors [32.02774117064752]
  We tackle the Extrapolated View Synthesis (EVS) problem by evaluating the reconstructions on views such as looking left, right or downwards. To the best of our knowledge, we are the first to address the EVS problem in urban scene reconstruction.
  arXiv  Detail & Related papers  (2024-07-03T09:23:13Z)
• A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose [44.13819148680788]
  We develop a novel construct-and-optimize method for sparse view synthesis without camera poses. Specifically, we construct a solution by using monocular depth and projecting pixels back into the 3D world. We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views.
  arXiv  Detail & Related papers  (2024-05-06T17:36:44Z)
• DUSt3R: Geometric 3D Vision Made Easy [8.471330244002564]
  We introduce DUSt3R, a novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera.
  arXiv  Detail & Related papers  (2023-12-21T18:52:14Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• Towards Scalable Multi-View Reconstruction of Geometry and Materials [27.660389147094715]
  We propose a novel method for joint recovery of camera pose, object geometry and spatially-varying Bidirectional Reflectance Distribution Function (svBRDF) of 3D scenes. The input are high-resolution RGBD images captured by a mobile, hand-held capture system with point lights for active illumination.
  arXiv  Detail & Related papers  (2023-06-06T15:07:39Z)
• Image Stabilization for Hololens Camera in Remote Collaboration [1.1470070927586016]
  Narrow field-of-view (FoV) and motion blur can offer an unpleasant experience with limited cognition for remote viewers of AR headsets. We propose a two-stage pipeline to tackle this issue and ensure a stable viewing experience with a larger FoV. The solution involves an offline 3D reconstruction of the indoor environment, followed by enhanced rendering using only the live poses of AR device.
  arXiv  Detail & Related papers  (2023-04-05T20:35:49Z)
• State of the Art in Dense Monocular Non-Rigid 3D Reconstruction [100.9586977875698]
  3D reconstruction of deformable (or non-rigid) scenes from a set of monocular 2D image observations is a long-standing and actively researched area of computer vision and graphics. This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views.
  arXiv  Detail & Related papers  (2022-10-27T17:59:53Z)
• Towards 3D Scene Reconstruction from Locally Scale-Aligned Monocular Video Depth [90.33296913575818]
  In some video-based scenarios such as video depth estimation and 3D scene reconstruction from a video, the unknown scale and shift residing in per-frame prediction may cause the depth inconsistency. We propose a locally weighted linear regression method to recover the scale and shift with very sparse anchor points. Our method can boost the performance of existing state-of-the-art approaches by 50% at most over several zero-shot benchmarks.
  arXiv  Detail & Related papers  (2022-02-03T08:52:54Z)
• Stable View Synthesis [100.86844680362196]
  We present Stable View Synthesis (SVS) Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets.
  arXiv  Detail & Related papers  (2020-11-14T07:24:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.