Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images

• URL: http://arxiv.org/abs/2412.06250v1
• Date: Mon, 09 Dec 2024 06:58:31 GMT
• Title: Splatter-360: Generalizable 360$^{\circ}$ Gaussian Splatting for Wide-baseline Panoramic Images
• Authors: Zheng Chen, Chenming Wu, Zhelun Shen, Chen Zhao, Weicai Ye, Haocheng Feng, Errui Ding, Song-Hai Zhang,
• Abstract summary: textitSplatter-360 is a novel end-to-end generalizable 3DGS framework to handle wide-baseline panoramic images.<n>We introduce a 3D-aware bi-projection encoder to mitigate the distortions inherent in panoramic images.<n>This enables robust 3D-aware feature representations and real-time rendering capabilities.
• Score: 52.48351378615057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Wide-baseline panoramic images are frequently used in applications like VR and simulations to minimize capturing labor costs and storage needs. However, synthesizing novel views from these panoramic images in real time remains a significant challenge, especially due to panoramic imagery's high resolution and inherent distortions. Although existing 3D Gaussian splatting (3DGS) methods can produce photo-realistic views under narrow baselines, they often overfit the training views when dealing with wide-baseline panoramic images due to the difficulty in learning precise geometry from sparse 360$^{\circ}$ views. This paper presents \textit{Splatter-360}, a novel end-to-end generalizable 3DGS framework designed to handle wide-baseline panoramic images. Unlike previous approaches, \textit{Splatter-360} performs multi-view matching directly in the spherical domain by constructing a spherical cost volume through a spherical sweep algorithm, enhancing the network's depth perception and geometry estimation. Additionally, we introduce a 3D-aware bi-projection encoder to mitigate the distortions inherent in panoramic images and integrate cross-view attention to improve feature interactions across multiple viewpoints. This enables robust 3D-aware feature representations and real-time rendering capabilities. Experimental results on the HM3D~\cite{hm3d} and Replica~\cite{replica} demonstrate that \textit{Splatter-360} significantly outperforms state-of-the-art NeRF and 3DGS methods (e.g., PanoGRF, MVSplat, DepthSplat, and HiSplat) in both synthesis quality and generalization performance for wide-baseline panoramic images. Code and trained models are available at \url{https://3d-aigc.github.io/Splatter-360/}.

Related papers

• You Need a Transition Plane: Bridging Continuous Panoramic 3D Reconstruction with Perspective Gaussian Splatting [57.44295803750027]
  We present a novel framework, named TPGS, to bridge continuous panoramic 3D scene reconstruction with perspective Gaussian splatting. Specifically, we optimize 3D Gaussians within individual cube faces and then fine-tune them in the stitched panoramic space. Experiments on indoor and outdoor, egocentric, and roaming benchmark datasets demonstrate that our approach outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-12T03:42:50Z)
• SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting [53.32467009064287]
  We propose a text-driven 3D-consistent scene generation model: SceneDreamer360. Our proposed method leverages a text-driven panoramic image generation model as a prior for 3D scene generation. Our experiments demonstrate that SceneDreamer360 with its panoramic image generation and 3DGS can produce higher quality, spatially consistent, and visually appealing 3D scenes from any text prompt.
  arXiv  Detail & Related papers  (2024-08-25T02:56:26Z)
• LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation [105.52153675890408]
  3D immersive scene generation is a challenging yet critical task in computer vision and graphics. Layerpano3D is a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt.
  arXiv  Detail & Related papers  (2024-08-23T17:50:23Z)
• DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting [56.101576795566324]
  We present a text-to-3D 360$circ$ scene generation pipeline. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement. Our method offers a globally consistent 3D scene within a 360$circ$ perspective.
  arXiv  Detail & Related papers  (2024-04-10T10:46:59Z)
• 360-GS: Layout-guided Panoramic Gaussian Splatting For Indoor Roaming [15.62029018680868]
  3D Gaussian Splatting (3D-GS) has attracted great attention with real-time and photo-realistic renderings. We propose 360-GS, a novel $360circ$ Gaussian splatting for a limited set of panoramic inputs. We show that 360-GS allows panoramic rendering and outperforms state-of-the-art methods with fewer artifacts in novel view synthesis.
  arXiv  Detail & Related papers  (2024-02-01T16:52:21Z)
• PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas [54.4948540627471]
  We propose PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas. Unlike generalizable radiance fields trained on perspective images, PanoGRF avoids the information loss from panorama-to-perspective conversion. Results on multiple panoramic datasets demonstrate that PanoGRF significantly outperforms state-of-the-art generalizable view synthesis methods.
  arXiv  Detail & Related papers  (2023-06-02T13:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.