Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding

• URL: http://arxiv.org/abs/2601.02339v1
• Date: Mon, 05 Jan 2026 18:33:50 GMT
• Title: Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding
• Abstract summary: We propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches.<n>Unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor to capture fine-grained 3D shape details.<n>We employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations.
• Score: 86.55824709875598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.

Related papers

• FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation [48.928051741750124]
  We propose a scalable image-to-3DGS framework that boosts representation learning and cross-modal alignment during generation.<n>FLUX3D yields substantial improvements in appearance fidelity and significantly outperforms all state-of-the-art (SOTA) methods in generating high-quality 3DGS assets.
  arXiv  Detail & Related papers  (2026-06-23T17:52:21Z)
• NRGS: Neural Regularization for Robust 3D Semantic Gaussian Splatting [41.21645092733966]
  We propose a neural regularization method that refines the noisy 3D semantic field produced by lifting multi-view inconsistent 2D features.<n> Experiments show that our method enhances the accuracy of lifted semantics, providing an efficient and effective approach to robust 3D semantic Gaussian Splatting.
  arXiv  Detail & Related papers  (2026-04-24T10:57:05Z)
• GS^2: Graph-based Spatial Distribution Optimization for Compact 3D Gaussian Splatting [50.2045409672451]
  3D Gaussian Splatting (3DGS) has demonstrated breakthrough performance in novel view synthesis and real-time rendering.<n>It is constrained by the high memory cost due to a huge number of Gaussian points.<n>We propose graph-based spatial distribution optimization for compact 3DGS.
  arXiv  Detail & Related papers  (2026-04-02T10:41:51Z)
• TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment [58.46706158310462]
  TIGaussian harnesses 3D Gaussian Splatting (3DGS) characteristics to strengthen cross-modality alignment.<n>Our multi-branch 3DGS tokenizer decouples the intrinsic properties of 3DGS structures into compact latent representations.<n>A text-3D projection module adaptively maps 3D features to text embedding space for better text-3D alignment.
  arXiv  Detail & Related papers  (2026-01-27T06:30:32Z)
• Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting [33.7339252839354]
  We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level.<n>Inspired by keypoint detection, our decoder learns to distribute primitives across image patches.<n>Our resulting pose-free model generates scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models.
  arXiv  Detail & Related papers  (2025-12-17T14:59:21Z)
• GSFF-SLAM: 3D Semantic Gaussian Splatting SLAM via Feature Field [17.57215792490409]
  GSFF-SLAM is a novel dense semantic SLAM system based on 3D Gaussian Splatting.<n>Our method supports semantic reconstruction using various forms of 2D priors, particularly sparse and noisy signals.<n>When utilizing 2D ground truth priors, GSFF-SLAM achieves state-of-the-art semantic segmentation performance with 95.03% mIoU.
  arXiv  Detail & Related papers  (2025-04-28T01:21:35Z)
• BG-Triangle: Bézier Gaussian Triangle for 3D Vectorization and Rendering [60.240908644910874]
  Differentiable rendering enables efficient optimization by allowing gradients to be computed through the rendering process.<n>Existing solutions approximate or re-formulate traditional rendering operations using smooth, probabilistic proxies.<n>We present a novel hybrid representation that combines B'ezier triangle-based vector graphics primitives with Gaussian-based probabilistic models.
  arXiv  Detail & Related papers  (2025-03-18T06:53:52Z)
• DirectTriGS: Triplane-based Gaussian Splatting Field Representation for 3D Generation [37.09199962653554]
  We present DirectTriGS, a novel framework designed for 3D object generation with Gaussian Splatting (GS)<n>The proposed generation framework can produce high-quality 3D object geometry and rendering results in the text-to-3D task.
  arXiv  Detail & Related papers  (2025-03-10T04:05:38Z)
• LinPrim: Linear Primitives for Differentiable Volumetric Rendering [51.56484100374058]
  We introduce two new scene representations based on linear primitives.<n>We present a different octaiableizer that runs efficiently on GPU.<n>We demonstrate comparable performance to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-27T18:49:38Z)
• Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding [59.51535163599723]
  FreeGS is an unsupervised semantic-embedded 3DGS framework that achieves view-consistent 3D scene understanding without the need for 2D labels.<n>FreeGS performs comparably to state-of-the-art methods while avoiding the complex data preprocessing workload.
  arXiv  Detail & Related papers  (2024-11-29T08:52:32Z)
• PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting [54.7468067660037]
  PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices.<n>Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS.
  arXiv  Detail & Related papers  (2024-10-29T15:28:15Z)
• GSplatLoc: Grounding Keypoint Descriptors into 3D Gaussian Splatting for Improved Visual Localization [1.4466437171584356]
  We propose a two-stage procedure that integrates dense and robust keypoint descriptors from the lightweight XFeat feature extractor into 3DGS.<n>In the second stage, the initial pose estimate is refined by minimizing the rendering-based photometric warp loss.<n> Benchmarking on widely used indoor and outdoor datasets demonstrates improvements over recent neural rendering-based localization methods.
  arXiv  Detail & Related papers  (2024-09-24T23:18:32Z)
• CLIP-GS: CLIP-Informed Gaussian Splatting for View-Consistent 3D Indoor Semantic Understanding [17.440124130814166]
  Exploiting 3D Gaussian Splatting (3DGS) with Contrastive Language-Image Pre-Training (CLIP) models for open-vocabulary 3D semantic understanding of indoor scenes has emerged as an attractive research focus.<n>We present CLIP-GS, efficiently achieving a coherent semantic understanding of 3D indoor scenes via the proposed Semantic Attribute Compactness (SAC) and 3D Coherent Regularization (3DCR)<n>Our method remarkably suppresses existing state-of-the-art approaches, achieving mIoU improvements of 21.20% and 13.05% on ScanNet and Replica datasets, respectively
  arXiv  Detail & Related papers  (2024-04-22T15:01:32Z)
• latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction [48.86083272054711]
  latentSplat is a method to predict semantic Gaussians in a 3D latent space that can be splatted and decoded by a light-weight generative 2D architecture. We show that latentSplat outperforms previous works in reconstruction quality and generalization, while being fast and scalable to high-resolution data.
  arXiv  Detail & Related papers  (2024-03-24T20:48:36Z)
• Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting [27.974762304763694]
  We introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features into a novel semantic component of 3D Gaussians. We build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference.
  arXiv  Detail & Related papers  (2024-03-22T21:28:19Z)
• Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering [71.44349029439944]
  Recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering.
  arXiv  Detail & Related papers  (2023-11-30T17:58:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.