Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields

• URL: http://arxiv.org/abs/2402.13252v1
• Date: Tue, 20 Feb 2024 18:59:02 GMT
• Title: Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields
• Authors: Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu
• Abstract summary: We propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor. We also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask.
• Score: 26.4340697184666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization.

Related papers

• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis [53.702118455883095]
  We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting. Our key idea lies in exploring the self-supervisions inherent in the binocular stereo consistency between each pair of binocular images. Our method significantly outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-24T15:10:27Z)
• Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS [52.3215552448623]
  Novel View Synthesis (NVS) without Structure-from-Motion (SfM) pre-processed camera poses are crucial for promoting rapid response capabilities and enhancing robustness against variable operating conditions. Recent SfM-free methods have integrated pose optimization, designing end-to-end frameworks for joint camera pose estimation and NVS. Most existing works rely on per-pixel image loss functions, such as L2 loss. In this study, we propose a correspondence-guided SfM-free 3D Gaussian splatting for NVS.
  arXiv  Detail & Related papers  (2024-08-16T13:11:22Z)
• GaussianRoom: Improving 3D Gaussian Splatting with SDF Guidance and Monocular Cues for Indoor Scene Reconstruction [3.043712258792239]
  We present a unified framework integrating neural SDF with 3DGS. This framework incorporates a learnable neural SDF field to guide the densification and pruning of Gaussians. Our method achieves state-of-the-art performance in both surface reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2024-05-30T03:46:59Z)
• GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time [112.32349668385635]
  GGRt is a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $ge$ 5 FPS and real-time rendering at $ge$ 100 FPS.
  arXiv  Detail & Related papers  (2024-03-15T09:47:35Z)
• GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis [70.24111297192057]
  We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner. The proposed method enables 2K-resolution rendering under a sparse-view camera setting.
  arXiv  Detail & Related papers  (2023-12-04T18:59:55Z)
• HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance [19.252300247300145]
  This work proposes holistic sampling and smoothing approaches to achieve high-quality text-to-3D generation. We compute denoising scores in the text-to-image diffusion model's latent and image spaces. To generate high-quality renderings in a single-stage optimization, we propose regularization for the variance of z-coordinates along NeRF rays.
  arXiv  Detail & Related papers  (2023-05-30T05:56:58Z)
• Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering [22.2095090385119]
  Given an imperfect reconstructed 3D model, most previous methods have focused on the refinement of either geometry, texture, or camera pose. We propose a novel optimization approach based on differentiable rendering, which integrates the optimization of camera pose, geometry, and texture into a unified framework. Using differentiable rendering, an image-level adversarial loss is applied to further improve the 3D model, making it more photorealistic.
  arXiv  Detail & Related papers  (2022-08-15T04:32:41Z)
• IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images [52.021529273866896]
  We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content. Our method adopts neural representations for geometry as signed distance fields (SDFs) and materials during optimization to enjoy their flexibility and compactness. We show that our IRON achieves significantly better inverse rendering quality compared to prior works.
  arXiv  Detail & Related papers  (2022-04-05T14:14:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.