Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion

• URL: http://arxiv.org/abs/2406.02972v3
• Date: Tue, 18 Jun 2024 06:06:14 GMT
• Title: Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion
• Authors: Tianyi Xiong, Jiayi Wu, Botao He, Cornelia Fermuller, Yiannis Aloimonos, Heng Huang, Christopher A. Metzler,
• Abstract summary: Event3DGS can reconstruct high-fidelity 3D structure and appearance under high-speed egomotion. Experiments on multiple synthetic and real-world datasets demonstrate the superiority of Event3DGS compared with existing event-based dense 3D scene reconstruction frameworks. Our framework also allows one to incorporate a few motion-blurred frame-based measurements into the reconstruction process to further improve appearance fidelity without loss of structural accuracy.
• Score: 54.197343533492486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By combining differentiable rendering with explicit point-based scene representations, 3D Gaussian Splatting (3DGS) has demonstrated breakthrough 3D reconstruction capabilities. However, to date 3DGS has had limited impact on robotics, where high-speed egomotion is pervasive: Egomotion introduces motion blur and leads to artifacts in existing frame-based 3DGS reconstruction methods. To address this challenge, we introduce Event3DGS, an {\em event-based} 3DGS framework. By exploiting the exceptional temporal resolution of event cameras, Event3GDS can reconstruct high-fidelity 3D structure and appearance under high-speed egomotion. Extensive experiments on multiple synthetic and real-world datasets demonstrate the superiority of Event3DGS compared with existing event-based dense 3D scene reconstruction frameworks; Event3DGS substantially improves reconstruction quality (+3dB) while reducing computational costs by 95\%. Our framework also allows one to incorporate a few motion-blurred frame-based measurements into the reconstruction process to further improve appearance fidelity without loss of structural accuracy.

Related papers

• E-3DGS: Gaussian Splatting with Exposure and Motion Events [29.042018288378447]
  We propose E-3DGS, a novel event-based approach that partitions events into motion and exposure. We introduce a novel integration of 3DGS with exposure events for high-quality reconstruction of explicit scene representations. Our method is faster and delivers better reconstruction quality than event-based NeRF while being more cost-effective than NeRF methods.
  arXiv  Detail & Related papers  (2024-10-22T13:17:20Z)
• EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting [76.02450110026747]
  Event cameras, inspired by biological vision, record pixel-wise intensity changes asynchronously with high temporal resolution. We propose Event-Aided Free-Trajectory 3DGS, which seamlessly integrates the advantages of event cameras into 3DGS. We evaluate our method on the public Tanks and Temples benchmark and a newly collected real-world dataset, RealEv-DAVIS.
  arXiv  Detail & Related papers  (2024-10-20T13:44:24Z)
• Elite-EvGS: Learning Event-based 3D Gaussian Splatting by Distilling Event-to-Video Priors [8.93657924734248]
  Event cameras are bio-inspired sensors that output asynchronous and sparse event streams, instead of fixed frames. We propose a novel event-based 3DGS framework, named Elite-EvGS. Our key idea is to distill the prior knowledge from the off-the-shelf event-to-video (E2V) models to effectively reconstruct 3D scenes from events.
  arXiv  Detail & Related papers  (2024-09-20T10:47:52Z)
• EaDeblur-GS: Event assisted 3D Deblur Reconstruction with Gaussian Splatting [8.842593320829785]
  Event-assisted 3D Deblur Reconstruction with Gaussian Splatting (EaDeblur-GS) is presented. It integrates event camera data to enhance the robustness of 3DGS against motion blur. It achieves sharp 3D reconstructions in real-time, demonstrating performance comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-18T13:55:54Z)
• WildGaussians: 3D Gaussian Splatting in the Wild [80.5209105383932]
  We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data.
  arXiv  Detail & Related papers  (2024-07-11T12:41:32Z)
• SpikeGS: Reconstruct 3D scene via fast-moving bio-inspired sensors [28.68263688378836]
  Spike Gausian Splatting (SpikeGS) is a framework that integrates spike streams into 3DGS pipeline to reconstruct 3D scenes via a fast-moving bio-inspired camera. SpikeGS extracts detailed geometry and texture from high temporal resolution but texture lacking spike stream, reconstructs 3D scenes captured in 1 second.
  arXiv  Detail & Related papers  (2024-07-04T09:32:12Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition [66.80822249039235]
  3D Gaussian Splatting has emerged as an alternative 3D representation for novel view synthesis. We propose SAGD, a conceptually simple yet effective boundary-enhanced segmentation pipeline for 3D-GS. Our approach achieves high-quality 3D segmentation without rough boundary issues, which can be easily applied to other scene editing tasks.
  arXiv  Detail & Related papers  (2024-01-31T14:19:03Z)
• EvAC3D: From Event-based Apparent Contours to 3D Models via Continuous Visual Hulls [46.94040300725127]
  3D reconstruction from multiple views is a successful computer vision field with multiple deployments in applications. We study the problem of 3D reconstruction from event-cameras, motivated by the advantages of event-based cameras in terms of low power and latency. We propose Apparent Contour Events (ACE), a novel event-based representation that defines the geometry of the apparent contour of an object.
  arXiv  Detail & Related papers  (2023-04-11T15:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.