Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras

• URL: http://arxiv.org/abs/2409.04751v2
• Date: Wed, 11 Sep 2024 07:04:58 GMT
• Title: Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras
• Authors: Zimu Liao, Siyan Chen, Rong Fu, Yi Wang, Zhongling Su, Hao Luo, Li Ma, Linning Xu, Bo Dai, Hengjie Li, Zhilin Pei, Xingcheng Zhang,
• Abstract summary: We introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods.
• Score: 17.75361921797657
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, 3D Gaussian Splatting (3DGS) has garnered attention for its high fidelity and real-time rendering. However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation. Additionally, there are inefficiencies in the tile-based splatting, especially for the extreme curvature and wide field of view of fisheye lenses, which are crucial for its broader real-life applications. To tackle these challenges, we introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design. Since we only modified the projection component, it can also be easily adapted for use with different camera models. Compared to methods that train after undistortion, our approach demonstrates a clear improvement in visual quality.

Related papers

• Exploring Surround-View Fisheye Camera 3D Object Detection [44.09164098759971]
  We investigate the technical feasibility of implementing end-to-end 3D object detection with surround-view fisheye camera system.<n>We develop two methods that incorporate the unique geometry of fisheye images into mainstream detection frameworks.<n> Experiments on Fisheye3DOD show that our fisheye-compatible modeling improves accuracy by up to 6.2% over baseline methods.
  arXiv  Detail & Related papers  (2025-11-24T02:28:56Z)
• PCR-GS: COLMAP-Free 3D Gaussian Splatting via Pose Co-Regularizations [102.0476991174456]
  COLMAP-free 3DGS has attracted increasing attention due to its remarkable performance in reconstructing high-quality 3D scenes from unposed images or videos.<n>We propose PCR-GS, an innovative COLMAP-free 3DGS technique that achieves superior 3D scene modeling and camera pose estimation via camera pose co-regularization.
  arXiv  Detail & Related papers  (2025-07-18T13:09:33Z)
• Systematic Comparison of Projection Methods for Monocular 3D Human Pose Estimation on Fisheye Images [8.719294151596705]
  We evaluate the impact of pinhole, equidistant and double sphere camera models, as well as cylindrical projection methods, on 3D human pose estimation accuracy.<n>We find that in close-up scenarios, pinhole projection is inadequate, and the optimal projection method varies with the FOV covered by the human pose.<n>We propose a for selecting the appropriate projection model based on the detection bounding box to enhance prediction quality.
  arXiv  Detail & Related papers  (2025-06-24T16:05:36Z)
• UniK3D: Universal Camera Monocular 3D Estimation [62.06785782635153]
  We present UniK3D, the first generalizable method for monocular 3D estimation able to model any camera. Our method introduces a spherical 3D representation which allows for better disentanglement of camera and scene geometry. A comprehensive zero-shot evaluation on 13 diverse datasets demonstrates the state-of-the-art performance of UniK3D across 3D, depth, and camera metrics.
  arXiv  Detail & Related papers  (2025-03-20T17:49:23Z)
• DoF-Gaussian: Controllable Depth-of-Field for 3D Gaussian Splatting [52.52398576505268]
  We introduce DoF-Gaussian, a controllable depth-of-field method for 3D-GS. We develop a lens-based imaging model based on geometric optics principles to control DoF effects. Our framework is customizable and supports various interactive applications.
  arXiv  Detail & Related papers  (2025-03-02T05:57:57Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision. Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data. We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• EVER: Exact Volumetric Ellipsoid Rendering for Real-time View Synthesis [72.53316783628803]
  We present Exact Volumetric Ellipsoid Rendering (EVER), a method for real-time differentiable emission-only volume rendering. Unlike recentization based approach by 3D Gaussian Splatting (3DGS), our primitive based representation allows for exact volume rendering. We show that our method is more accurate with blending issues than 3DGS and follow-up work on view rendering.
  arXiv  Detail & Related papers  (2024-10-02T17:59:09Z)
• DOF-GS: Adjustable Depth-of-Field 3D Gaussian Splatting for Refocusing,Defocus Rendering and Blur Removal [42.427021878005405]
  3D Gaussian Splatting techniques have recently advanced 3D scene reconstruction and novel view synthesis, achieving high-quality real-time rendering. These approaches are inherently limited by the underlying pinhole camera assumption in modeling the images and hence only work for All-in-Focus (AiF) sharp image inputs. This severely affects their applicability in real-world scenarios where images often exhibit defocus blur due to the limited depth-of-field (DOF) of imaging devices. We introduce DOF-GS that allows for rendering adjustable DOF effects, removing defocus blur as well as refocusing of 3D scenes,
  arXiv  Detail & Related papers  (2024-05-27T16:54:49Z)
• Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion [25.54868552979793]
  We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods.
  arXiv  Detail & Related papers  (2024-03-20T06:19:41Z)
• VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction [59.40711222096875]
  We present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets.
  arXiv  Detail & Related papers  (2024-02-27T11:40:50Z)
• Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting [55.71424195454963]
  Spec-Gaussian is an approach that utilizes an anisotropic spherical Gaussian appearance field instead of spherical harmonics. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces.
  arXiv  Detail & Related papers  (2024-02-24T17:22:15Z)
• Gaussian Splatting SLAM [16.3858380078553]
  We present the first application of 3D Gaussian Splatting in monocular SLAM. Our method runs live at 3fps, unifying the required representation for accurate tracking, mapping, and high-quality rendering. Several innovations are required to continuously reconstruct 3D scenes with high fidelity from a live camera.
  arXiv  Detail & Related papers  (2023-12-11T18:19:04Z)
• Neural Lens Modeling [50.57409162437732]
  NeuroLens is a neural lens model for distortion and vignetting that can be used for point projection and ray casting. It can be used to perform pre-capture calibration using classical calibration targets, and can later be used to perform calibration or refinement during 3D reconstruction. The model generalizes across many lens types and is trivial to integrate into existing 3D reconstruction and rendering systems.
  arXiv  Detail & Related papers  (2023-04-10T20:09:17Z)
• A Novel Viewport-Adaptive Motion Compensation Technique for Fisheye Video [7.09875977818162]
  A recently proposed projection-based approach takes the fisheye projection into account to improve fisheye motion compensation. We propose a novel viewport-adaptive motion compensation technique that applies the motion vectors in different perspective viewports. We achieve average gains of +2.40 dB in terms of PSNR compared to the state of the art in fisheye motion compensation.
  arXiv  Detail & Related papers  (2022-02-28T15:41:08Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• Redesigning SLAM for Arbitrary Multi-Camera Systems [51.81798192085111]
  Adding more cameras to SLAM systems improves robustness and accuracy but complicates the design of the visual front-end significantly. In this work, we aim at an adaptive SLAM system that works for arbitrary multi-camera setups. We adapt a state-of-the-art visual-inertial odometry with these modifications, and experimental results show that the modified pipeline can adapt to a wide range of camera setups.
  arXiv  Detail & Related papers  (2020-03-04T11:44:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.