COLMAP-Free 3D Gaussian Splatting
- URL: http://arxiv.org/abs/2312.07504v1
- Date: Tue, 12 Dec 2023 18:39:52 GMT
- Title: COLMAP-Free 3D Gaussian Splatting
- Authors: Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros,
Xiaolong Wang
- Abstract summary: We propose a novel method to perform novel view synthesis without any SfM preprocessing.
We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time.
Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes.
- Score: 93.69157280273856
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While neural rendering has led to impressive advances in scene reconstruction
and novel view synthesis, it relies heavily on accurately pre-computed camera
poses. To relax this constraint, multiple efforts have been made to train
Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the
implicit representations of NeRFs provide extra challenges to optimize the 3D
structure and camera poses at the same time. On the other hand, the recently
proposed 3D Gaussian Splatting provides new opportunities given its explicit
point cloud representations. This paper leverages both the explicit geometric
representation and the continuity of the input video stream to perform novel
view synthesis without any SfM preprocessing. We process the input frames in a
sequential manner and progressively grow the 3D Gaussians set by taking one
input frame at a time, without the need to pre-compute the camera poses. Our
method significantly improves over previous approaches in view synthesis and
camera pose estimation under large motion changes. Our project page is
https://oasisyang.github.io/colmap-free-3dgs
Related papers
- A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose [44.13819148680788]
We develop a novel construct-and-optimize method for sparse view synthesis without camera poses.
Specifically, we construct a solution by using monocular depth and projecting pixels back into the 3D world.
We demonstrate results on the Tanks and Temples and Static Hikes datasets with as few as three widely-spaced views.
arXiv Detail & Related papers (2024-05-06T17:36:44Z) - Denoising Diffusion via Image-Based Rendering [54.20828696348574]
We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes.
First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes.
Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
arXiv Detail & Related papers (2024-02-05T19:00:45Z) - Free3D: Consistent Novel View Synthesis without 3D Representation [63.931920010054064]
Free3D is a simple accurate method for monocular open-set novel view synthesis (NVS)
Compared to other works that took a similar approach, we obtain significant improvements without resorting to an explicit 3D representation.
arXiv Detail & Related papers (2023-12-07T18:59:18Z) - Pose-Free Generalizable Rendering Transformer [72.47072706742065]
PF-GRT is a Pose-Free framework for Generalizable Rendering Transformer.
PF-GRT is parameterized using a local relative coordinate system.
Experiments with zero-shot rendering on datasets reveal that it produces superior quality in generating photo-realistic images.
arXiv Detail & Related papers (2023-10-05T17:24:36Z) - LEAP: Liberate Sparse-view 3D Modeling from Camera Poses [28.571234973474077]
We present LEAP, a pose-free approach for sparse-view 3D modeling.
LEAP discards pose-based operations and learns geometric knowledge from data.
We show LEAP significantly outperforms prior methods when they employ predicted poses from state-of-the-art pose estimators.
arXiv Detail & Related papers (2023-10-02T17:59:37Z) - BAA-NGP: Bundle-Adjusting Accelerated Neural Graphics Primitives [6.431806897364565]
Implicit neural representations have become pivotal in robotic perception, enabling robots to comprehend 3D environments from 2D images.
We propose a framework called bundle-adjusting accelerated neural graphics primitives (BAA-NGP)
Results demonstrate 10 to 20 x speed improvement compared to other bundle-adjusting neural radiance field methods.
arXiv Detail & Related papers (2023-06-07T05:36:45Z) - FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses
via Pixel-Aligned Scene Flow [26.528667940013598]
Reconstruction of 3D neural fields from posed images has emerged as a promising method for self-supervised representation learning.
Key challenge preventing the deployment of these 3D scene learners on large-scale video data is their dependence on precise camera poses from structure-from-motion.
We propose a method that jointly reconstructs camera poses and 3D neural scene representations online and in a single forward pass.
arXiv Detail & Related papers (2023-05-31T20:58:46Z) - Structure-Aware NeRF without Posed Camera via Epipolar Constraint [8.115535686311249]
The neural radiance field (NeRF) for realistic novel view synthesis requires camera poses to be pre-acquired.
We integrate the pose extraction and view synthesis into a single end-to-end procedure so they can benefit from each other.
arXiv Detail & Related papers (2022-10-01T03:57:39Z) - CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields [67.76151996543588]
We learn a 3D- and camera-aware generative model which faithfully recovers not only the image but also the camera data distribution.
At test time, our model generates images with explicit control over the camera as well as the shape and appearance of the scene.
arXiv Detail & Related papers (2021-03-31T17:59:24Z) - GNeRF: GAN-based Neural Radiance Field without Posed Camera [67.80805274569354]
We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field reconstruction for the complex scenarios with unknown and even randomly camera poses.
Our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.
arXiv Detail & Related papers (2021-03-29T13:36:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.