Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation

• URL: http://arxiv.org/abs/2604.10951v1
• Date: Mon, 13 Apr 2026 03:49:06 GMT
• Title: Fast-SegSim: Real-Time Open-Vocabulary Segmentation for Robotics in Simulation
• Abstract summary: Fast-SegSim is a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting.<n>Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation.<n>Fast-SegSim provides critical value in robotic applications, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels.
• Score: 23.703731324592656
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal navigation, successfully doubling the navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.

Related papers

• Stipple: Real-Time Incremental Gaussian Splatting with Visual-Inertial Tracking [45.88028371034407]
  We propose a new method that tracks and reconstructs simultaneously in real-time by leveraging an efficient visual-inertial tracking system.<n>We show that many of the heavy preprocessing and training steps of 3DGS can be replaced with a more efficient incremental training strategy.
  arXiv  Detail & Related papers  (2026-08-02T02:17:05Z)
• Ψ-Map: Panoptic Surface Integrated Mapping Enables Real2Sim Transfer [24.777932568749446]
  We propose a comprehensive framework that integrates geometric reinforcement, end-to-end panoptic learning, and efficient rendering.<n>Our system achieves superior geometric and panoptic reconstruction quality in large-scale scenes while maintaining an inference rate exceeding 40 FPS.
  arXiv  Detail & Related papers  (2026-04-13T04:41:00Z)
• Mobile-GS: Real-time Gaussian Splatting for Mobile Devices [16.605749198745105]
  3D Gaussian Splatting (3DGS) has emerged as a powerful representation for high-quality rendering across a wide range of applications.<n>We propose a mobile-tailored real-time Gaussian Splatting method, dubbed Mobile-GS, enabling efficient inference of Gaussian Splatting on edge devices.<n>Our proposed Mobile-GS achieves real-time rendering and compact model size while preserving high visual quality, making it well-suited for mobile applications.
  arXiv  Detail & Related papers  (2026-03-12T04:33:04Z)
• FTSplat: Feed-forward Triangle Splatting Network [0.0]
  We propose a feed-forward framework for triangle primitive generation that directly predicts continuous triangle surfaces from calibrated multi-view images.<n>Our method produces simulation-ready models in a single forward pass, obviating the need for per-scene optimization or post-processing.
  arXiv  Detail & Related papers  (2026-03-06T06:04:09Z)
• Quantile Rendering: Efficiently Embedding High-dimensional Feature on 3D Gaussian Splatting [52.18697134979677]
  Recent advancements in computer vision have successfully extended Open-vocabulary segmentation (OVS) to the 3D domain by leveraging 3D Gaussian Splatting (3D-GS)<n>Existing methods employ codebooks or feature compression, causing information loss, thereby degrading segmentation quality.<n>We introduce Quantile Rendering (Q-Render), a novel rendering strategy for 3D Gaussians that efficiently handles high-dimensional features while maintaining high fidelity.<n>Our framework outperforms state-of-the-art methods, while enabling real-time rendering with an approximate 43.7x speedup on 512-D feature maps.
  arXiv  Detail & Related papers  (2025-12-24T04:16:18Z)
• SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation [114.57192386025373]
  SegSplat is a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding.<n>This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments.
  arXiv  Detail & Related papers  (2025-11-23T10:26:38Z)
• Perceive-Sample-Compress: Towards Real-Time 3D Gaussian Splatting [7.421996491601524]
  We introduce a novel perceive-sample-compress framework for 3D Gaussian Splatting.<n>We show that our method significantly improves memory efficiency and high visual quality while maintaining real-time rendering speed.
  arXiv  Detail & Related papers  (2025-08-07T01:34:38Z)
• From Coarse to Fine: Learnable Discrete Wavelet Transforms for Efficient 3D Gaussian Splatting [5.026688852582894]
  AutoOpti3DGS is a training-time framework that automatically restrains Gaussian proliferation without sacrificing visual fidelity.<n>Wavelet-driven, coarse-to-fine process delays the formation of redundant fine Gaussians.
  arXiv  Detail & Related papers  (2025-06-29T00:27:17Z)
• LODGE: Level-of-Detail Large-Scale Gaussian Splatting with Efficient Rendering [75.67501939005119]
  We present a novel level-of-detail (LOD) method for 3D Gaussian Splatting on memory-constrained devices.<n>Our approach iteratively selects optimal subsets of Gaussians based on camera distance.<n>Our method achieves state-of-the-art performance on both outdoor (Hierarchical 3DGS) and indoor (Zip-NeRF) datasets.
  arXiv  Detail & Related papers  (2025-05-29T06:50:57Z)
• QuickSplat: Fast 3D Surface Reconstruction via Learned Gaussian Initialization [69.50126552763157]
  Surface reconstruction is fundamental to computer vision and graphics, enabling applications in 3D modeling, mixed reality, robotics, and more.<n>Existing approaches based on rendering obtain promising results, but optimize on a per-scene basis, resulting in a slow optimization that can struggle to model textureless regions.<n>We introduce QuickSplat, which learns data-driven priors to generate dense initializations for 2D gaussian splatting optimization of large-scale indoor scenes.
  arXiv  Detail & Related papers  (2025-05-08T18:43:26Z)
• MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo [54.00987996368157]
  We present MVSGaussian, a new generalizable 3D Gaussian representation approach derived from Multi-View Stereo (MVS) MVSGaussian achieves real-time rendering with better synthesis quality for each scene.
  arXiv  Detail & Related papers  (2024-05-20T17:59:30Z)
• GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering [112.16239342037714]
  GES (Generalized Exponential Splatting) is a novel representation that employs Generalized Exponential Function (GEF) to model 3D scenes. With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks.
  arXiv  Detail & Related papers  (2024-02-15T17:32:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.