UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations

• URL: http://arxiv.org/abs/2411.15355v1
• Date: Fri, 22 Nov 2024 21:59:46 GMT
• Title: UniGaussian: Driving Scene Reconstruction from Multiple Camera Models via Unified Gaussian Representations
• Abstract summary: How to effectively simulate fisheye cameras in driving scene remains an unsolved problem. We propose UniGaussian, a novel approach that learns a unified 3D Gaussian representation from multiple camera models. Our approach models multiple sensors (pinhole and fisheye cameras) and modalities (depth, semantic, normal and LiDAR point clouds)
• Score: 23.346765633086452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Urban scene reconstruction is crucial for real-world autonomous driving simulators. Although existing methods have achieved photorealistic reconstruction, they mostly focus on pinhole cameras and neglect fisheye cameras. In fact, how to effectively simulate fisheye cameras in driving scene remains an unsolved problem. In this work, we propose UniGaussian, a novel approach that learns a unified 3D Gaussian representation from multiple camera models for urban scene reconstruction in autonomous driving. Our contributions are two-fold. First, we propose a new differentiable rendering method that distorts 3D Gaussians using a series of affine transformations tailored to fisheye camera models. This addresses the compatibility issue of 3D Gaussian splatting with fisheye cameras, which is hindered by light ray distortion caused by lenses or mirrors. Besides, our method maintains real-time rendering while ensuring differentiability. Second, built on the differentiable rendering method, we design a new framework that learns a unified Gaussian representation from multiple camera models. By applying affine transformations to adapt different camera models and regularizing the shared Gaussians with supervision from different modalities, our framework learns a unified 3D Gaussian representation with input data from multiple sources and achieves holistic driving scene understanding. As a result, our approach models multiple sensors (pinhole and fisheye cameras) and modalities (depth, semantic, normal and LiDAR point clouds). Our experiments show that our method achieves superior rendering quality and fast rendering speed for driving scene simulation.

Related papers

• ConeGaussian: Anti-Aliased Gaussian Ray-Tracing for Generic Central Cameras [56.46810752790842]
  We present ConeGaussian, a camera-model-agnostic anti-aliasing framework for Gaussian ray-based rendering.<n>We derive a closed-form response under a locally linear, depth-local, moment-matched approximation of the finite pixel footprint.<n>Unlike in mip-splatting, our scene-space frequency floor and filtering enable trivial composition at render time, allowing us to remove excess blurring.
  arXiv  Detail & Related papers  (2026-09-11T18:05:07Z)
• iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion [62.09575122593993]
  iGaussian is a two-stage feed-forward framework that achieves real-time camera pose estimation through direct 3D Gaussian inversion.<n> Experimental results on the NeRF Synthetic, Mip-NeRF 360, and T&T+DB datasets demonstrate a significant performance improvement over previous methods.
  arXiv  Detail & Related papers  (2025-11-18T05:22:22Z)
• GaussRender: Learning 3D Occupancy with Gaussian Rendering [86.89653628311565]
  GaussRender is a module that improves 3D occupancy learning by enforcing projective consistency. Our method penalizes 3D configurations that produce inconsistent 2D projections, thereby enforcing a more coherent 3D structure.
  arXiv  Detail & Related papers  (2025-02-07T16:07:51Z)
• EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splatting [22.590036750925627]
  Photorealistic reconstruction of street scenes is essential for developing real-world simulators in autonomous driving. Recent methods based on 3D/4D Gaussian Splatting (GS) have demonstrated promising results, but they still encounter challenges in complex street scenes due to the unpredictable motion of dynamic objects. We propose Explicit Motion Decomposition (EMD), which models the motions of dynamic objects by introducing learnable motion embeddings to the Gaussians.
  arXiv  Detail & Related papers  (2024-11-23T15:10:04Z)
• GPS-Gaussian+: Generalizable Pixel-wise 3D Gaussian Splatting for Real-Time Human-Scene Rendering from Sparse Views [67.34073368933814]
  We propose a generalizable Gaussian Splatting approach for high-resolution image rendering under a sparse-view camera setting. We train our Gaussian parameter regression module on human-only data or human-scene data, jointly with a depth estimation module to lift 2D parameter maps to 3D space. Experiments on several datasets demonstrate that our method outperforms state-of-the-art methods while achieving an exceeding rendering speed.
  arXiv  Detail & Related papers  (2024-11-18T08:18:44Z)
• No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images [100.80376573969045]
  NoPoSplat is a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from multi-view images. Our model achieves real-time 3D Gaussian reconstruction during inference. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-31T17:58:22Z)
• UniGS: Modeling Unitary 3D Gaussians for Novel View Synthesis from Sparse-view Images [20.089890859122168]
  We introduce UniGS, a novel 3D Gaussian reconstruction and novel view synthesis model. UniGS predicts a high-fidelity representation of 3D Gaussians from arbitrary number of posed sparse-view images.
  arXiv  Detail & Related papers  (2024-10-17T03:48:02Z)
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos [58.22272760132996]
  We show that existing 4D Gaussian methods dramatically fail in this setup because the monocular setting is underconstrained. We propose Dynamic Gaussian Marbles, which consist of three core modifications that target the difficulties of the monocular setting. We evaluate on the Nvidia Dynamic Scenes dataset and the DyCheck iPhone dataset, and show that Gaussian Marbles significantly outperforms other Gaussian baselines in quality.
  arXiv  Detail & Related papers  (2024-06-26T19:37:07Z)
• DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos [23.967885144420613]
  We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. Our method achieves state-of-the-art performance in novel view synthesis.
  arXiv  Detail & Related papers  (2024-05-27T23:38:10Z)
• Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion [20.464224937528222]
  Differentiable 3D-Gaussian splatting (GS) is emerging as a prominent technique in computer vision and graphics for reconstructing 3D scenes. GS suffers from a well-known'missing cone' problem, which results in poor reconstruction along the depth axis. We propose fusion algorithms that simultaneously utilize RGB camera data and sonar data.
  arXiv  Detail & Related papers  (2024-04-06T17:23:43Z)
• Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting [27.974762304763694]
  We introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features into a novel semantic component of 3D Gaussians. We build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference.
  arXiv  Detail & Related papers  (2024-03-22T21:28:19Z)
• BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting [8.380954205255104]
  BAD-Gaussians is a novel approach to handle severe motion-blurred images with inaccurate camera poses. Our method achieves superior rendering quality compared to previous state-of-the-art deblur neural rendering methods.
  arXiv  Detail & Related papers  (2024-03-18T14:43:04Z)
• DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes [57.12439406121721]
  We present DrivingGaussian, an efficient and effective framework for surrounding dynamic autonomous driving scenes. For complex scenes with moving objects, we first sequentially and progressively model the static background of the entire scene. We then leverage a composite dynamic Gaussian graph to handle multiple moving objects. We further use a LiDAR prior for Gaussian Splatting to reconstruct scenes with greater details and maintain panoramic consistency.
  arXiv  Detail & Related papers  (2023-12-13T06:30:51Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.