RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS

• URL: http://arxiv.org/abs/2403.13806v1
• Date: Wed, 20 Mar 2024 17:59:55 GMT
• Title: RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS
• Authors: Michael Niemeyer, Fabian Manhardt, Marie-Julie Rakotosaona, Michael Oechsle, Daniel Duckworth, Rama Gosula, Keisuke Tateno, John Bates, Dominik Kaeser, Federico Tombari,
• Abstract summary: We present RadSplat, a lightweight method for robust real-time rendering of complex scenes. First, we use radiance fields as a prior and supervision signal for optimizing point-based scene representations, leading to improved quality and more robust optimization. Next, we develop a novel pruning technique reducing the overall point count while maintaining high quality, leading to smaller and more compact scene representations with faster inference speeds.
• Score: 47.47003067842151
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in view synthesis and real-time rendering have achieved photorealistic quality at impressive rendering speeds. While Radiance Field-based methods achieve state-of-the-art quality in challenging scenarios such as in-the-wild captures and large-scale scenes, they often suffer from excessively high compute requirements linked to volumetric rendering. Gaussian Splatting-based methods, on the other hand, rely on rasterization and naturally achieve real-time rendering but suffer from brittle optimization heuristics that underperform on more challenging scenes. In this work, we present RadSplat, a lightweight method for robust real-time rendering of complex scenes. Our main contributions are threefold. First, we use radiance fields as a prior and supervision signal for optimizing point-based scene representations, leading to improved quality and more robust optimization. Next, we develop a novel pruning technique reducing the overall point count while maintaining high quality, leading to smaller and more compact scene representations with faster inference speeds. Finally, we propose a novel test-time filtering approach that further accelerates rendering and allows to scale to larger, house-sized scenes. We find that our method enables state-of-the-art synthesis of complex captures at 900+ FPS.

Related papers

• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
  We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
  arXiv  Detail & Related papers  (2024-02-27T11:40:50Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)
• Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction [29.83056271799794]
  Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction and rendering. We propose a deformable 3D Gaussians Splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space. Through a differential Gaussianizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed.
  arXiv  Detail & Related papers  (2023-09-22T16:04:02Z)
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering [4.320393382724066]
  We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
  arXiv  Detail & Related papers  (2023-08-08T06:37:06Z)
• Progressively Optimized Local Radiance Fields for Robust View Synthesis [76.55036080270347]
  We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window.
  arXiv  Detail & Related papers  (2023-03-24T04:03:55Z)
• BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis [42.93055827628597]
  We present a method for reconstructing high-quality meshes of large real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians.
  arXiv  Detail & Related papers  (2023-02-28T18:58:03Z)
• PDRF: Progressively Deblurring Radiance Field for Fast and Robust Scene Reconstruction from Blurry Images [75.87721926918874]
  We present Progressively Deblurring Radiance Field (PDRF) PDRF is a novel approach to efficiently reconstruct high quality radiance fields from blurry images. We show that PDRF is 15X faster than previous State-of-The-Art scene reconstruction methods.
  arXiv  Detail & Related papers  (2022-08-17T03:42:29Z)
• NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction [50.54946139497575]
  We propose NeRFusion, a method that combines the advantages of NeRF and TSDF-based fusion techniques to achieve efficient large-scale reconstruction and photo-realistic rendering. We demonstrate that NeRFusion achieves state-of-the-art quality on both large-scale indoor and small-scale object scenes, with substantially faster reconstruction than NeRF and other recent methods.
  arXiv  Detail & Related papers  (2022-03-21T18:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.