GaussFusion: Towards Multimodal 3D Gaussian Pretraining

• URL: http://arxiv.org/abs/2607.05906v1
• Date: Tue, 07 Jul 2026 07:01:46 GMT
• Title: GaussFusion: Towards Multimodal 3D Gaussian Pretraining
• Abstract summary: GaussFusion is a multimodal pre-training framework for 3D Gaussian representations.<n>GaussFusion integrates image and text supervision into masked Gaussian modeling.<n>GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61% and 3.85%, respectively.
• Score: 22.9925711988588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D representation learning. Existing pre-training methods for Gaussian representations, such as masked Gaussian reconstruction, primarily capture local structures but offer limited semantic supervision. In this paper, we propose GaussFusion, a multimodal pre-training framework for 3D Gaussian representations. GaussFusion integrates image and text supervision into masked Gaussian modeling through cross-modal semantic alignment, enabling the Gaussian encoder to learn both visual and language-level semantic information during pre-training. To better adapt masked modeling to the non-uniform distribution of Gaussian primitives, we further propose Gaussian Salience-guided Multi-scale Hole Masking (GSHM). GSHM constructs spatially continuous masked regions based on Gaussian salience. By applying hole masks at multiple scales, GSHM encourages the encoder to capture both fine-grained local patterns and broader structural dependencies. Extensive experiments on downstream tasks demonstrate that GaussFusion improves the transferability of Gaussian representations. Notably, GaussFusion outperforms Gaussian-MAE on ModelNet40 and ScanObjectNN (PB-T50-RS) by 0.61\% and 3.85\%, respectively.

Related papers

• GaussianFusion: Unified 3D Gaussian Representation for Multi-Modal Fusion Perception [13.265767966811312]
  We propose a new universal framework for multi-modal fusion based on 3D Gaussian representation.<n>This approach naturally unifies multi-modal features within a shared and continuous 3D Gaussian space.<n>Experiments demonstrate the generality and robustness of GaussianFusion.
  arXiv  Detail & Related papers  (2026-07-01T10:28:15Z)
• Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction [72.85652875805387]
  Free-Range Gaussians is a multi-view reconstruction method that predicts non-pixel, non-voxel-aligned 3D Gaussians from as few as four images.<n>Our generative formulation of reconstruction allows the model to be supervised with non-grid-aligned 3D data.
  arXiv  Detail & Related papers  (2026-04-06T17:24:18Z)
• F4Splat: Feed-Forward Predictive Densification for Feed-Forward 3D Gaussian Splatting [28.800888081472422]
  We present F4Splat, which performs Feed-Forward predictive densification for Feed-Forward 3D Gaussian Splatting.<n>Our model predicts per-region densification scores to estimate the required Gaussian density and allows explicit control over the final Gaussian budget.<n>Experiments demonstrate that our model achieves superior novel-view synthesis performance compared to prior uncalibrated feed-forward methods.
  arXiv  Detail & Related papers  (2026-03-22T16:03:56Z)
• ShelfGaussian: Shelf-Supervised Open-Vocabulary Gaussian-based 3D Scene Understanding [7.610505486431266]
  We introduce ShelfGaussian, an open-vocabulary multi-modal Gaussian-based 3D scene understanding framework supervised by off-the-shelf vision foundation models.<n>Existing methods either model objects as closed-set semantic Gaussians supervised by annotated 3D labels, neglecting their rendering ability, or learn open-set Gaussian representations via purely 2D self-supervision.
  arXiv  Detail & Related papers  (2025-12-03T02:06:09Z)
• ProtoGS: Efficient and High-Quality Rendering with 3D Gaussian Prototypes [81.48624894781257]
  3D Gaussian Splatting (3DGS) has made significant strides in novel view synthesis but is limited by the substantial number of Gaussian primitives required.<n>Recent methods address this issue by compressing the storage size of densified Gaussians, yet fail to preserve rendering quality and efficiency.<n>We propose ProtoGS to learn Gaussian prototypes to represent Gaussian primitives, significantly reducing the total Gaussian amount without sacrificing visual quality.
  arXiv  Detail & Related papers  (2025-03-21T18:55:14Z)
• Gaussian Graph Network: Learning Efficient and Generalizable Gaussian Representations from Multi-view Images [12.274418254425019]
  3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis performance.<n>We propose Gaussian Graph Network (GGN) to generate efficient and generalizable Gaussian representations.<n>We conduct experiments on the large-scale RealEstate10K and ACID datasets to demonstrate the efficiency and generalization of our method.
  arXiv  Detail & Related papers  (2025-03-20T16:56:13Z)
• GaussianFormer-2: Probabilistic Gaussian Superposition for Efficient 3D Occupancy Prediction [55.60972844777044]
  3D semantic occupancy prediction is an important task for robust vision-centric autonomous driving.<n>Most existing methods leverage dense grid-based scene representations, overlooking the spatial sparsity of the driving scenes.<n>We propose a probabilistic Gaussian superposition model which interprets each Gaussian as a probability distribution of its neighborhood being occupied.
  arXiv  Detail & Related papers  (2024-12-05T17:59:58Z)
• PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views [116.10577967146762]
  PixelGaussian is an efficient framework for learning generalizable 3D Gaussian reconstruction from arbitrary views. Our method achieves state-of-the-art performance with good generalization to various numbers of views.
  arXiv  Detail & Related papers  (2024-10-24T17:59:58Z)
• MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering [61.64903786502728]
  We propose a novel approach that integrates mesh representation with 3D Gaussian splats to perform high-quality rendering of reconstructed real-world scenes. We consider the distance between each Gaussian splat and the mesh surface to distinguish between tightly-bound and loosely-bound splats. Our method surpasses recent mesh-based neural rendering techniques by achieving a 2dB higher PSNR, and outperforms mesh-based Gaussian splatting methods by 1.3 dB PSNR.
  arXiv  Detail & Related papers  (2024-10-11T16:07:59Z)
• GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling [55.05713977022407]
  We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting. Experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D all show that our model synthesis achieves state-of-the-art generation results.
  arXiv  Detail & Related papers  (2024-03-28T17:59:50Z)
• Mesh-based Gaussian Splatting for Real-time Large-scale Deformation [58.18290393082119]
  It is challenging for users to directly deform or manipulate implicit representations with large deformations in the real-time fashion. We develop a novel GS-based method that enables interactive deformation. Our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate.
  arXiv  Detail & Related papers  (2024-02-07T12:36:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.