GS4: Generalizable Sparse Splatting Semantic SLAM

• URL: http://arxiv.org/abs/2506.06517v1
• Date: Fri, 06 Jun 2025 20:28:37 GMT
• Title: GS4: Generalizable Sparse Splatting Semantic SLAM
• Authors: Mingqi Jiang, Chanho Kim, Chen Ziwen, Li Fuxin,
• Abstract summary: We introduce the first generalizable GS-based semantic SLAM algorithm incrementally builds and updates a 3D scene representation from an RGB-D video stream using a learned generalizable network.<n>We demonstrate state-of-the-art semantic SLAM performance on the real-world benchmark ScanNet with an order of magnitude fewer Gaussians compared to other recent GS-based methods.
• Score: 1.6024406587122393
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional SLAM algorithms are excellent at camera tracking but might generate lower resolution and incomplete 3D maps. Recently, Gaussian Splatting (GS) approaches have emerged as an option for SLAM with accurate, dense 3D map building. However, existing GS-based SLAM methods rely on per-scene optimization which is time-consuming and does not generalize to diverse scenes well. In this work, we introduce the first generalizable GS-based semantic SLAM algorithm that incrementally builds and updates a 3D scene representation from an RGB-D video stream using a learned generalizable network. Our approach starts from an RGB-D image recognition backbone to predict the Gaussian parameters from every downsampled and backprojected image location. Additionally, we seamlessly integrate 3D semantic segmentation into our GS framework, bridging 3D mapping and recognition through a shared backbone. To correct localization drifting and floaters, we propose to optimize the GS for only 1 iteration following global localization. We demonstrate state-of-the-art semantic SLAM performance on the real-world benchmark ScanNet with an order of magnitude fewer Gaussians compared to other recent GS-based methods, and showcase our model's generalization capability through zero-shot transfer to the NYUv2 and TUM RGB-D datasets.

Related papers

• RGB-Only Gaussian Splatting SLAM for Unbounded Outdoor Scenes [12.150995604820443]
  3D Gaussian Splatting (3DGS) has become a popular solution in SLAM, as it can produce high-fidelity novel views.<n>Previous GS-based methods primarily target indoor scenes and rely on RGB-D sensors or pre-trained depth estimation models.<n>We propose a RGB-only gaussian splatting SLAM method for unbounded outdoor scenes--OpenGS-SLAM.
  arXiv  Detail & Related papers  (2025-02-21T18:02:31Z)
• PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM [105.01907579424362]
  PanoSLAM is the first SLAM system to integrate geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation within a unified framework.<n>For the first time, it achieves panoptic 3D reconstruction of open-world environments directly from the RGB-D video.
  arXiv  Detail & Related papers  (2024-12-31T08:58:10Z)
• OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies [112.80292725951921]
  textbfOVGaussian is a generalizable textbfOpen-textbfVocabulary 3D semantic segmentation framework based on the 3D textbfGaussian representation.<n>We first construct a large-scale 3D scene dataset based on 3DGS, dubbed textbfSegGaussian, which provides detailed semantic and instance annotations for both Gaussian points and multi-view images.<n>To promote semantic generalization across scenes, we introduce Generalizable Semantic Rasterization (GSR), which leverages a
  arXiv  Detail & Related papers  (2024-12-31T07:55:35Z)
• HI-SLAM2: Geometry-Aware Gaussian SLAM for Fast Monocular Scene Reconstruction [38.47566815670662]
  HI-SLAM2 is a geometry-aware Gaussian SLAM system that achieves fast and accurate monocular scene reconstruction using only RGB input.<n>We demonstrate significant improvements over existing Neural SLAM methods and even surpass RGB-D-based methods in both reconstruction and rendering quality.
  arXiv  Detail & Related papers  (2024-11-27T01:39:21Z)
• IG-SLAM: Instant Gaussian SLAM [6.228980850646457]
  3D Gaussian Splatting has recently shown promising results as an alternative scene representation in SLAM systems. We present IG-SLAM, a dense RGB-only SLAM system that employs robust Dense-SLAM methods for tracking and combines them with Gaussian Splatting. We demonstrate competitive performance with state-of-the-art RGB-only SLAM systems while achieving faster operation speeds.
  arXiv  Detail & Related papers  (2024-08-02T09:07:31Z)
• Splat-SLAM: Globally Optimized RGB-only SLAM with 3D Gaussians [87.48403838439391]
  3D Splatting has emerged as a powerful representation of geometry and appearance for RGB-only dense Simultaneous SLAM. We propose the first RGB-only SLAM system with a dense 3D Gaussian map representation. Our experiments on the Replica, TUM-RGBD, and ScanNet datasets indicate the effectiveness of globally optimized 3D Gaussians.
  arXiv  Detail & Related papers  (2024-05-26T12:26:54Z)
• GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data [73.06536202251915]
  3D Shape represented as point cloud has achieve advancements in multimodal pre-training to align image and language descriptions. We propose GS-CLIP for the first attempt to introduce 3DGS into multimodal pre-training to enhance 3D representation.
  arXiv  Detail & Related papers  (2024-02-09T05:46:47Z)
• SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM [48.190398577764284]
  SplaTAM is an approach to enable high-fidelity reconstruction from a single unposed RGB-D camera. It employs a simple online tracking and mapping system tailored to the underlying Gaussian representation. Experiments show that SplaTAM achieves up to 2x superior performance in camera pose estimation, map construction, and novel-view synthesis over existing methods.
  arXiv  Detail & Related papers  (2023-12-04T18:53:24Z)
• GS-IR: 3D Gaussian Splatting for Inverse Rendering [71.14234327414086]
  We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) We extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering.
  arXiv  Detail & Related papers  (2023-11-26T02:35:09Z)
• GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting [51.96353586773191]
  We introduce textbfGS-SLAM that first utilizes 3D Gaussian representation in the Simultaneous Localization and Mapping system. Our method utilizes a real-time differentiable splatting rendering pipeline that offers significant speedup to map optimization and RGB-D rendering. Our method achieves competitive performance compared with existing state-of-the-art real-time methods on the Replica, TUM-RGBD datasets.
  arXiv  Detail & Related papers  (2023-11-20T12:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.