iComMa: Inverting 3D Gaussian Splatting for Camera Pose Estimation via Comparing and Matching

• URL: http://arxiv.org/abs/2312.09031v2
• Date: Wed, 20 Mar 2024 12:00:59 GMT
• Title: iComMa: Inverting 3D Gaussian Splatting for Camera Pose Estimation via Comparing and Matching
• Authors: Yuan Sun, Xuan Wang, Yunfan Zhang, Jie Zhang, Caigui Jiang, Yu Guo, Fei Wang,
• Abstract summary: We present a method named iComMa to address the 6D camera pose estimation problem in computer vision. We propose an efficient method for accurate camera pose estimation by inverting 3D Gaussian Splatting (3DGS)
• Score: 14.737266480464156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a method named iComMa to address the 6D camera pose estimation problem in computer vision. Conventional pose estimation methods typically rely on the target's CAD model or necessitate specific network training tailored to particular object classes. Some existing methods have achieved promising results in mesh-free object and scene pose estimation by inverting the Neural Radiance Fields (NeRF). However, they still struggle with adverse initializations such as large rotations and translations. To address this issue, we propose an efficient method for accurate camera pose estimation by inverting 3D Gaussian Splatting (3DGS). Specifically, a gradient-based differentiable framework optimizes camera pose by minimizing the residual between the query image and the rendered image, requiring no training. An end-to-end matching module is designed to enhance the model's robustness against adverse initializations, while minimizing pixel-level comparing loss aids in precise pose estimation. Experimental results on synthetic and complex real-world data demonstrate the effectiveness of the proposed approach in challenging conditions and the accuracy of camera pose estimation.

Related papers

• No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images [100.80376573969045]
  NoPoSplat is a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from multi-view images. Our model achieves real-time 3D Gaussian reconstruction during inference. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-31T17:58:22Z)
• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• Correspondence-Guided SfM-Free 3D Gaussian Splatting for NVS [52.3215552448623]
  Novel View Synthesis (NVS) without Structure-from-Motion (SfM) pre-processed camera poses are crucial for promoting rapid response capabilities and enhancing robustness against variable operating conditions. Recent SfM-free methods have integrated pose optimization, designing end-to-end frameworks for joint camera pose estimation and NVS. Most existing works rely on per-pixel image loss functions, such as L2 loss. In this study, we propose a correspondence-guided SfM-free 3D Gaussian splatting for NVS.
  arXiv  Detail & Related papers  (2024-08-16T13:11:22Z)
• Cameras as Rays: Pose Estimation via Ray Diffusion [54.098613859015856]
  Estimating camera poses is a fundamental task for 3D reconstruction and remains challenging given sparsely sampled views. We propose a distributed representation of camera pose that treats a camera as a bundle of rays. Our proposed methods, both regression- and diffusion-based, demonstrate state-of-the-art performance on camera pose estimation on CO3D.
  arXiv  Detail & Related papers  (2024-02-22T18:59:56Z)
• ContraNeRF: 3D-Aware Generative Model via Contrastive Learning with Unsupervised Implicit Pose Embedding [40.36882490080341]
  We propose a novel 3D-aware GAN optimization technique through contrastive learning with implicit pose embeddings. We make the discriminator estimate a high-dimensional implicit pose embedding from a given image and perform contrastive learning on the pose embedding. The proposed approach can be employed for the dataset, where the canonical camera pose is ill-defined because it does not look up or estimate camera poses.
  arXiv  Detail & Related papers  (2023-04-27T07:53:13Z)
• RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimization [46.144194562841435]
  We propose a framework based on a recurrent neural network (RNN) for object pose refinement. The problem is formulated as a non-linear least squares problem based on the estimated correspondence field. The correspondence field estimation and pose refinement are conducted alternatively in each iteration to recover accurate object poses.
  arXiv  Detail & Related papers  (2022-03-24T06:24:55Z)
• Camera Distortion-aware 3D Human Pose Estimation in Video with Optimization-based Meta-Learning [23.200130129530653]
  Existing 3D human pose estimation algorithms trained on distortion-free datasets suffer performance drop when applied to new scenarios with a specific camera distortion. We propose a simple yet effective model for 3D human pose estimation in video that can quickly adapt to any distortion environment.
  arXiv  Detail & Related papers  (2021-11-30T01:35:04Z)
• Self-Supervised 3D Hand Pose Estimation from monocular RGB via Contrastive Learning [50.007445752513625]
  We propose a new self-supervised method for the structured regression task of 3D hand pose estimation. We experimentally investigate the impact of invariant and equivariant contrastive objectives. We show that a standard ResNet-152, trained on additional unlabeled data, attains an improvement of $7.6%$ in PA-EPE on FreiHAND.
  arXiv  Detail & Related papers  (2021-06-10T17:48:57Z)
• Wide-Baseline Relative Camera Pose Estimation with Directional Learning [46.21836501895394]
  We introduce DirectionNet, which estimates discrete distributions over the 5D relative pose space using a novel parameterization to make the estimation problem tractable. We evaluate our model on challenging synthetic and real pose estimation datasets constructed from Matterport3D and InteriorNet.
  arXiv  Detail & Related papers  (2021-06-07T04:46:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.