Online Language Splatting

• URL: http://arxiv.org/abs/2503.09447v3
• Date: Thu, 25 Sep 2025 00:40:54 GMT
• Title: Online Language Splatting
• Authors: Saimouli Katragadda, Cho-Ying Wu, Yuliang Guo, Xinyu Huang, Guoquan Huang, Liu Ren,
• Abstract summary: We introduce Online Language Splatting, the first framework to achieve online, near real-time, open-vocabulary language mapping within a 3DGS-SLAM system.<n>We show that our online method surpasses the state-of-the-art offline methods in accuracy and achieves more than 40x efficiency boost.
• Score: 28.066910888210973
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To enable AI agents to interact seamlessly with both humans and 3D environments, they must not only perceive the 3D world accurately but also align human language with 3D spatial representations. While prior work has made significant progress by integrating language features into geometrically detailed 3D scene representations using 3D Gaussian Splatting (GS), these approaches rely on computationally intensive offline preprocessing of language features for each input image, limiting adaptability to new environments. In this work, we introduce Online Language Splatting, the first framework to achieve online, near real-time, open-vocabulary language mapping within a 3DGS-SLAM system without requiring pre-generated language features. The key challenge lies in efficiently fusing high-dimensional language features into 3D representations while balancing the computation speed, memory usage, rendering quality and open-vocabulary capability. To this end, we innovatively design: (1) a high-resolution CLIP embedding module capable of generating detailed language feature maps in 18ms per frame, (2) a two-stage online auto-encoder that compresses 768-dimensional CLIP features to 15 dimensions while preserving open-vocabulary capabilities, and (3) a color-language disentangled optimization approach to improve rendering quality. Experimental results show that our online method not only surpasses the state-of-the-art offline methods in accuracy but also achieves more than 40x efficiency boost, demonstrating the potential for dynamic and interactive AI applications.

Related papers

• EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding [66.80528512321106]
  EmbodiedSplat is an online feed-forward 3DGS for open-vocabulary scene understanding.<n>Our objectives are 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner, and 2) Highly generalizable to novel scenes with feed-forward design.
  arXiv  Detail & Related papers  (2026-03-04T16:40:41Z)
• LEGO-SLAM: Language-Embedded Gaussian Optimization SLAM [2.0524609401792397]
  We propose LEGO-SLAM, a framework to achieve real-time, open-vocabulary mapping within a 3DGS-based SLAM system.<n>At the core of our method is a scene-adaptive encoder-decoder that distills high-dimensional language embeddings into a compact 16-dimensional feature space.<n>Experiments demonstrate that LEGO-SLAM achieves competitive mapping quality and tracking accuracy, all while providing open-vocabulary capabilities at 15 FPS.
  arXiv  Detail & Related papers  (2025-11-20T08:31:34Z)
• Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression [0.0]
  We introduce Gen-LangSplat, that replaces the scene-wise autoencoder with a generalized autoencoder, pre-trained extensively on the large-scale ScanNet dataset.<n>This architectural shift enables the use of a fixed, compact latent space for language features across any new scene without any scene-specific training.<n>Our results demonstrate that generalized embeddings can efficiently and accurately support open-vocabulary querying in novel 3D scenes.
  arXiv  Detail & Related papers  (2025-10-27T02:13:38Z)
• GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting [74.56128224977279]
  We present GALA, a novel framework for open-vocabulary 3D scene understanding with 3D Gaussian Splatting (3DGS)<n>GALA distills a scene-specific 3D instance feature field via self-supervised contrastive learning.<n>It supports seamless 2D and 3D open-vocabulary queries and reduces memory consumption by avoiding per-Gaussian high-dimensional feature learning.
  arXiv  Detail & Related papers  (2025-08-19T21:26:49Z)
• GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond [56.677984098204696]
  multimodal language models are driving the development of 3D Vision-Language Models (VLMs)<n>We propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations.<n>We present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images.
  arXiv  Detail & Related papers  (2025-07-01T15:52:59Z)
• 3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation [17.294440057314812]
  Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks.<n>We propose Geometric Distillation, a framework that injects human-inspired geometric cues into pretrained VLMs.<n>Our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs.
  arXiv  Detail & Related papers  (2025-06-11T15:56:59Z)
• 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models [58.80200897869225]
  We propose 4D LangSplat, which learns 4D language fields to handle time-agnostic or time-sensitive open-vocabulary queries in dynamic scenes efficiently. 4D LangSplat bypasses learning the language field from vision features and instead learns directly from text generated from object-wise video captions. Our results demonstrate that 4D LangSplat attains precise and efficient results for both time-sensitive and time-agnostic open-vocabulary queries.
  arXiv  Detail & Related papers  (2025-03-13T14:58:22Z)
• Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration [41.046653227409564]
  Dr. Splat is a novel approach for open-vocabulary 3D scene understanding leveraging 3D Gaussian Splatting. Our method associates language-aligned CLIP embeddings with 3D Gaussians for holistic 3D scene understanding. Experiments demonstrate that our approach significantly outperforms existing approaches in 3D perception benchmarks.
  arXiv  Detail & Related papers  (2025-02-23T17:01:14Z)
• ChatSplat: 3D Conversational Gaussian Splatting [51.40403199909113]
  ChatSplat is a system that constructs a 3D language field, enabling rich chat-based interaction within 3D space.<n>For view-level interaction, we designed an encoder that encodes the rendered feature map of each view into tokens, which are then processed by a large language model.<n>At the scene level, ChatSplat combines multi-view tokens, enabling interactions that consider the entire scene.
  arXiv  Detail & Related papers  (2024-12-01T08:59:30Z)
• g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks [62.74304008688472]
  Generalizable 3D-Language Feature Fields (g3D-LF) is a 3D representation model pre-trained on large-scale 3D-language dataset for embodied tasks.
  arXiv  Detail & Related papers  (2024-11-26T01:54:52Z)
• 4-LEGS: 4D Language Embedded Gaussian Splatting [12.699978393733309]
  We show how to lift-temporal features to a 4D representation based on 3D Gaussianting.<n>This enables an interactive interface where the user cantemporally localize events in the video from text prompts.<n>We demonstrate our system on public 3D video datasets of people and animals performing various actions.
  arXiv  Detail & Related papers  (2024-10-14T17:00:53Z)
• O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation [9.431926560072412]
  We propose O2V-mapping, which utilizes voxel-based language and geometric features to create an open-vocabulary field. Experiments on open-vocabulary object localization and semantic segmentation demonstrate that O2V-mapping achieves online construction of language scenes.
  arXiv  Detail & Related papers  (2024-04-10T08:54:43Z)
• FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding [11.118857208538039]
  We present Foundation Model Embedded Gaussian Splatting (S), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS) Results demonstrate remarkable multi-view semantic consistency, facilitating diverse downstream tasks, beating state-of-the-art methods by 10.2 percent on open-vocabulary language-based object detection. This research explores the intersection of vision, language, and 3D scene representation, paving the way for enhanced scene understanding in uncontrolled real-world environments.
  arXiv  Detail & Related papers  (2024-01-03T20:39:02Z)
• GPT4Point: A Unified Framework for Point-Language Understanding and Generation [76.61439685940272]
  GPT4Point is a groundbreaking point-language multimodal model for unified 3D object understanding and generation within the MLLM framework. GPT4Point as a powerful 3D MLLM seamlessly can execute a variety of point-text reference tasks such as point-cloud captioning and Q&A. It can get high-quality results through a low-quality point-text feature maintaining the geometric shapes and colors.
  arXiv  Detail & Related papers  (2023-12-05T18:59:55Z)
• Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding [2.517953665531978]
  We introduce Language Embedded 3D Gaussians, a novel scene representation for open-vocabulary query tasks. Our representation achieves the best visual quality and language querying accuracy across current language-embedded representations.
  arXiv  Detail & Related papers  (2023-11-30T11:50:07Z)
• ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction [62.599588577671796]
  We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality.
  arXiv  Detail & Related papers  (2023-11-29T20:30:18Z)
• RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding [46.253711788685536]
  We introduce a 3D-aware SFusion strategy that fuses 3D vision-language pairs derived from multiple 2D foundation models. We devise a region-aware point-discriminative contrastive learning objective to enable robust and effective 3D learning. Our model outperforms prior 3D open-world scene understanding approaches by an average of 17.2% and 9.1% for semantic and instance segmentation.
  arXiv  Detail & Related papers  (2023-04-03T13:30:04Z)
• PLA: Language-Driven Open-Vocabulary 3D Scene Understanding [57.47315482494805]
  Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. Recent breakthrough of 2D open-vocabulary perception is driven by Internet-scale paired image-text data with rich vocabulary concepts. We propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D.
  arXiv  Detail & Related papers  (2022-11-29T15:52:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.