GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation

• URL: http://arxiv.org/abs/2603.26661v1
• Date: Fri, 27 Mar 2026 17:58:05 GMT
• Title: GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation
• Abstract summary: We introduce a transformer-based model that directly generates 3D Gaussians via next-token prediction.<n>The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding.<n>Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons.
• Score: 42.49842620609683
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.

Related papers

• Generative 3D Gaussians with Learned Density Control [41.893507326094415]
  We present Density-Sampled Gaussians (DeG), a novel 3D representation designed to bridge the gap between adaptive rendering primitives and scalable generative modeling.
  arXiv  Detail & Related papers  (2026-05-08T17:54:25Z)
• TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens [25.44390249542703]
  We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal.<n>We propose to directly regress 3D mean coordinates using only a self-supervised rendering loss.<n>Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies.
  arXiv  Detail & Related papers  (2026-04-16T17:12:37Z)
• Free-Range Gaussians: Non-Grid-Aligned Generative 3D Gaussian Reconstruction [72.85652875805387]
  Free-Range Gaussians is a multi-view reconstruction method that predicts non-pixel, non-voxel-aligned 3D Gaussians from as few as four images.<n>Our generative formulation of reconstruction allows the model to be supervised with non-grid-aligned 3D data.
  arXiv  Detail & Related papers  (2026-04-06T17:24:18Z)
• Distilling Multi-view Diffusion Models into 3D Generators [4.3238419212557115]
  We introduce DD3G, a formulation that Distills a multi-view Diffusion model (MV-DM) into a 3D Generator using gaussian splatting.<n> DD3G compresses and integrates extensive visual and spatial knowledge from the MV-DM.<n>We propose PEPD, a generator consisting of Pattern Extraction and Progressive Decoding phases, which enables efficient fusion of probabilistic flow.
  arXiv  Detail & Related papers  (2025-04-01T06:32:48Z)
• GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding [44.68350305790145]
  GaussTR is a novel Transformer framework that unifies sparse 3D modeling with foundation model alignment through Gaussian representations to advance 3D spatial understanding.<n>Experiments on the Occ3D-nuScenes dataset demonstrate GaussTR's state-of-the-art zero-shot performance of 12.27 mIoU, along with a 40% reduction in training time.<n>These results highlight the efficacy of GaussTR for scalable and holistic 3D spatial understanding, with promising implications in autonomous driving and embodied agents.
  arXiv  Detail & Related papers  (2024-12-17T18:59:46Z)
• L3DG: Latent 3D Gaussian Diffusion [74.36431175937285]
  L3DG is the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. We employ a sparse convolutional architecture to efficiently operate on room-scale scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time.
  arXiv  Detail & Related papers  (2024-10-17T13:19:32Z)
• Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes [50.92217884840301]
  Gaussian Opacity Fields (GOF) is a novel approach for efficient, high-quality, and adaptive surface reconstruction in scenes. GOF is derived from ray-tracing-based volume rendering of 3D Gaussians. GOF surpasses existing 3DGS-based methods in surface reconstruction and novel view synthesis.
  arXiv  Detail & Related papers  (2024-04-16T17:57:19Z)
• GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling [55.05713977022407]
  We introduce a radiance representation that is both structured and fully explicit and thus greatly facilitates 3D generative modeling. We derive GaussianCube by first using a novel densification-constrained Gaussian fitting algorithm, which yields high-accuracy fitting. Experiments conducted on unconditional and class-conditioned object generation, digital avatar creation, and text-to-3D all show that our model synthesis achieves state-of-the-art generation results.
  arXiv  Detail & Related papers  (2024-03-28T17:59:50Z)
• Mesh-based Gaussian Splatting for Real-time Large-scale Deformation [58.18290393082119]
  It is challenging for users to directly deform or manipulate implicit representations with large deformations in the real-time fashion. We develop a novel GS-based method that enables interactive deformation. Our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate.
  arXiv  Detail & Related papers  (2024-02-07T12:36:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.