SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion

• URL: http://arxiv.org/abs/2407.10062v1
• Date: Sun, 14 Jul 2024 03:19:30 GMT
• Title: SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion
• Authors: Jiyuan Zhang, Kang Chen, Shiyan Chen, Yajing Zheng, Tiejun Huang, Zhaofei Yu,
• Abstract summary: Novel View Synthesis plays a crucial role by generating new 2D renderings from multi-view images of 3D scenes. High-frame-rate dense 3D reconstruction emerges as a vital technique, enabling detailed and accurate modeling of real-world objects or scenes. Spike cameras, a novel type of neuromorphic sensor, continuously record scenes with an ultra-high temporal resolution.
• Score: 46.23575738669567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Novel View Synthesis plays a crucial role by generating new 2D renderings from multi-view images of 3D scenes. However, capturing high-speed scenes with conventional cameras often leads to motion blur, hindering the effectiveness of 3D reconstruction. To address this challenge, high-frame-rate dense 3D reconstruction emerges as a vital technique, enabling detailed and accurate modeling of real-world objects or scenes in various fields, including Virtual Reality or embodied AI. Spike cameras, a novel type of neuromorphic sensor, continuously record scenes with an ultra-high temporal resolution, showing potential for accurate 3D reconstruction. Despite their promise, existing approaches, such as applying Neural Radiance Fields (NeRF) to spike cameras, encounter challenges due to the time-consuming rendering process. To address this issue, we make the first attempt to introduce the 3D Gaussian Splatting (3DGS) into spike cameras in high-speed capture, providing 3DGS as dense and continuous clues of views, then constructing SpikeGS. Specifically, to train SpikeGS, we establish computational equations between the rendering process of 3DGS and the processes of instantaneous imaging and exposing-like imaging of the continuous spike stream. Besides, we build a very lightweight but effective mapping process from spikes to instant images to support training. Furthermore, we introduced a new spike-based 3D rendering dataset for validation. Extensive experiments have demonstrated our method possesses the high quality of novel view rendering, proving the tremendous potential of spike cameras in modeling 3D scenes.

Related papers

• SweepEvGS: Event-Based 3D Gaussian Splatting for Macro and Micro Radiance Field Rendering from a Single Sweep [48.34647667445792]
  SweepEvGS is a novel hardware-integrated method that leverages event cameras for robust and accurate novel view synthesis from a single sweep. We validate the robustness and efficiency of SweepEvGS through experiments in three different imaging settings. Our results demonstrate that SweepEvGS surpasses existing methods in visual rendering quality, rendering speed, and computational efficiency.
  arXiv  Detail & Related papers  (2024-12-16T09:09:42Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision. Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data. We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• Speedy-Splat: Fast 3D Gaussian Splatting with Sparse Pixels and Sparse Primitives [60.217580865237835]
  3D Gaussian Splatting (3D-GS) is a recent 3D scene reconstruction technique that enables real-time rendering of novel views by modeling scenes as parametric point clouds of differentiable 3D Gaussians. We identify and address two key inefficiencies in 3D-GS, achieving substantial improvements in rendering speed, model size, and training time. Our Speedy-Splat approach combines these techniques to accelerate average rendering speed by a drastic $6.71times$ across scenes from the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets with $10.6times$ fewer primitives than 3
  arXiv  Detail & Related papers  (2024-11-30T20:25:56Z)
• SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream [20.552076533208687]
  A spike camera is a specialized high-speed visual sensor that offers advantages such as high temporal resolution and high dynamic range. We introduce SpikeGS, the method to learn 3D Gaussian fields solely from spike stream. Our method can reconstruct view synthesis results with fine texture details from a continuous spike stream captured by a moving spike camera.
  arXiv  Detail & Related papers  (2024-09-23T16:28:41Z)
• SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors [25.51366779254847]
  Spike Gausian Splatting (SpikeGS) is a framework that integrates spike streams into 3DGS pipeline to reconstruct 3D scenes via a fast-moving bio-inspired camera. SpikeGS extracts detailed geometry and texture from high temporal resolution but texture lacking spike stream, reconstructs 3D scenes captured in 1 second.
  arXiv  Detail & Related papers  (2024-07-04T09:32:12Z)
• Bootstrap 3D Reconstructed Scenes from 3D Gaussian Splatting [10.06208115191838]
  We present a bootstrapping method to enhance the rendering of novel views using trained 3D-GS. Our results indicate that bootstrapping effectively reduces artifacts, as well as clear enhancements on the evaluation metrics.
  arXiv  Detail & Related papers  (2024-04-29T12:57:05Z)
• Spike-NeRF: Neural Radiance Field Based On Spike Camera [24.829344089740303]
  We propose Spike-NeRF, the first Neural Radiance Field derived from spike data. Instead of the multi-view images at the same time of NeRF, the inputs of Spike-NeRF are continuous spike streams captured by a moving spike camera in a very short time. Our results demonstrate that Spike-NeRF produces more visually appealing results than the existing methods and the baseline we proposed in high-speed scenes.
  arXiv  Detail & Related papers  (2024-03-25T04:05:23Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.