VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points

• URL: http://arxiv.org/abs/2410.17932v2
• Date: Tue, 15 Apr 2025 11:59:15 GMT
• Title: VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points
• Authors: Linus Franke, Laura Fink, Marc Stamminger,
• Abstract summary: We propose a novel hybrid approach that combines the strengths of both point rendering directions regarding performance sweet spots.<n>For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output.<n>Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration.
• Score: 4.962171160815189
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in novel view synthesis have demonstrated impressive results in fast photorealistic scene rendering through differentiable point rendering, either via Gaussian Splatting (3DGS) [Kerbl and Kopanas et al. 2023] or neural point rendering [Aliev et al. 2020]. Unfortunately, these directions require either a large number of small Gaussians or expensive per-pixel post-processing for reconstructing fine details, which negatively impacts rendering performance. To meet the high performance demands of virtual reality (VR) systems, primitive or pixel counts therefore must be kept low, affecting visual quality. In this paper, we propose a novel hybrid approach based on foveated rendering as a promising solution that combines the strengths of both point rendering directions regarding performance sweet spots. Analyzing the compatibility with the human visual system, we find that using a low-detailed, few primitive smooth Gaussian representation for the periphery is cheap to compute and meets the perceptual demands of peripheral vision. For the fovea only, we use neural points with a convolutional neural network for the small pixel footprint, which provides sharp, detailed output within the rendering budget. This combination also allows for synergistic method accelerations with point occlusion culling and reducing the demands on the neural network. Our evaluation confirms that our approach increases sharpness and details compared to a standard VR-ready 3DGS configuration, and participants of a user study overwhelmingly preferred our method. Our system meets the necessary performance requirements for real-time VR interactions, ultimately enhancing the user's immersive experience. The project page can be found at: https://lfranke.github.io/vr_splatting

Related papers

• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization. We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars [19.249226899376943]
  We present SqueezeMe, a framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation. We achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset.
  arXiv  Detail & Related papers  (2024-12-19T18:46:55Z)
• 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes [87.01284850604495]
  We introduce 3D Convexting (3DCS), which leverages 3D smooth convexes as primitives for modeling geometrically-meaningful radiance fields from multiview images. 3DCS achieves superior performance over 3DGS on benchmarks such as MipNeizer, Tanks and Temples, and Deep Blending. Our results highlight the potential of 3D Convexting to become the new standard for high-quality scene reconstruction.
  arXiv  Detail & Related papers  (2024-11-22T14:31:39Z)
• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views [67.34073368933814]
  We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
  arXiv  Detail & Related papers  (2024-11-18T08:18:44Z)
• Few-shot Novel View Synthesis using Depth Aware 3D Gaussian Splatting [0.0]
  3D Gaussian splatting has surpassed neural radiance field methods in novel view synthesis. It produces a high-quality rendering with a lot of input views, but its performance drops significantly when only a few views are available. We propose a depth-aware Gaussian splatting method for few-shot novel view synthesis.
  arXiv  Detail & Related papers  (2024-10-14T20:42:30Z)
• N-BVH: Neural ray queries with bounding volume hierarchies [51.430495562430565]
  In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures. We devise N-BVH, a neural compression architecture designed to answer arbitrary ray queries in 3D. Our method provides faithful approximations of visibility, depth, and appearance attributes.
  arXiv  Detail & Related papers  (2024-05-25T13:54:34Z)
• OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering [55.50438181721271]
  Previous method utilizing NeRF for surface rendering to recover the occluded areas requires more than one day to train and several seconds to render occluded areas. We propose OccGaussian based on 3D Gaussian Splatting, which can be trained within 6 minutes and produces high-quality human renderings up to 160 FPS with occluded input.
  arXiv  Detail & Related papers  (2024-04-12T13:00:06Z)
• VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality [39.53150683721031]
  Our proposed VR-GS system represents a leap forward in human-centered 3D content interaction. The components of our Virtual Reality system are designed for high efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-01-30T01:28:36Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)
• VR-NeRF: High-Fidelity Virtualized Walkable Spaces [55.51127858816994]
  We present an end-to-end system for the high-fidelity capture, model reconstruction, and real-time rendering of walkable spaces in virtual reality using neural radiance fields.
  arXiv  Detail & Related papers  (2023-11-05T02:03:14Z)
• ABLE-NeRF: Attention-Based Rendering with Learnable Embeddings for Neural Radiance Field [20.986012773294714]
  We present an alternative to the physics-based VR approach by introducing a self-attention-based framework on volumes along a ray. Our method, which we call ABLE-NeRF, significantly reduces blurry' glossy surfaces in rendering and produces realistic translucent surfaces which lack in prior art.
  arXiv  Detail & Related papers  (2023-03-24T05:34:39Z)
• Immersive Neural Graphics Primitives [13.48024951446282]
  We present and evaluate a NeRF-based framework that is capable of rendering scenes in immersive VR. Our approach can yield a frame rate of 30 frames per second with a resolution of 1280x720 pixels per eye.
  arXiv  Detail & Related papers  (2022-11-24T09:33:38Z)
• Human Performance Modeling and Rendering via Neural Animated Mesh [40.25449482006199]
  We bridge the traditional mesh with a new class of neural rendering. In this paper, we present a novel approach for rendering human views from video. We demonstrate our approach on various platforms, inserting virtual human performances into AR headsets.
  arXiv  Detail & Related papers  (2022-09-18T03:58:00Z)
• NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing [34.40837543752915]
  We present a neural volumography technique called neural volumetric video or NeuVV to support immersive, interactive, and spatial-temporal rendering. NeuVV encodes a dynamic neural radiance field (NeRF) into renderable and editable primitives. We further develop a hybrid neural-rasterization rendering framework to support consumer-level VR headsets.
  arXiv  Detail & Related papers  (2022-02-12T15:23:16Z)
• HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars [65.82222842213577]
  We propose a novel neural rendering pipeline, which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. We then leverage the encoded information on the UV manifold to construct a 3D volumetric representation.
  arXiv  Detail & Related papers  (2021-12-19T17:34:15Z)
• Foveated Neural Radiance Fields for Real-Time and Egocentric Virtual Reality [11.969281058344581]
  High-quality 3D graphics requires large volumes of fine-detailed scene data for rendering. Recent approaches to combat this problem include remote rendering/streaming and neural representations of 3D assets. We present the first gaze-contingent 3D neural representation and view synthesis method.
  arXiv  Detail & Related papers  (2021-03-30T14:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.