2D Gaussians Meet Visual Tokenizer

• URL: http://arxiv.org/abs/2508.13515v2
• Date: Wed, 20 Aug 2025 01:19:13 GMT
• Title: 2D Gaussians Meet Visual Tokenizer
• Authors: Yiang Shi, Xiaoyang Guo, Wei Yin, Mingkai Jia, Qian Zhang, Xiaolin Hu, Wenyu Liu, Xinggang Wang,
• Abstract summary: Existing quantization-based tokenizers such as VQ-GAN primarily focus on appearance features like texture and color.<n>We propose Visual Gaussian Quantization (VGQ), a novel tokenizer paradigm that explicitly enhances structural modeling.<n>On the ImageNet 256x256 benchmark, VGQ achieves strong reconstruction quality with an rFID score of 1.00.
• Score: 46.20437041493538
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The image tokenizer is a critical component in AR image generation, as it determines how rich and structured visual content is encoded into compact representations. Existing quantization-based tokenizers such as VQ-GAN primarily focus on appearance features like texture and color, often neglecting geometric structures due to their patch-based design. In this work, we explored how to incorporate more visual information into the tokenizer and proposed a new framework named Visual Gaussian Quantization (VGQ), a novel tokenizer paradigm that explicitly enhances structural modeling by integrating 2D Gaussians into traditional visual codebook quantization frameworks. Our approach addresses the inherent limitations of naive quantization methods such as VQ-GAN, which struggle to model structured visual information due to their patch-based design and emphasis on texture and color. In contrast, VGQ encodes image latents as 2D Gaussian distributions, effectively capturing geometric and spatial structures by directly modeling structure-related parameters such as position, rotation and scale. We further demonstrate that increasing the density of 2D Gaussians within the tokens leads to significant gains in reconstruction fidelity, providing a flexible trade-off between token efficiency and visual richness. On the ImageNet 256x256 benchmark, VGQ achieves strong reconstruction quality with an rFID score of 1.00. Furthermore, by increasing the density of 2D Gaussians within the tokens, VGQ gains a significant boost in reconstruction capability and achieves a state-of-the-art reconstruction rFID score of 0.556 and a PSNR of 24.93, substantially outperforming existing methods. Codes will be released soon.

Related papers

• Fast 2DGS: Efficient Image Representation with Deep Gaussian Prior [21.89104780995278]
  Fast-2DGS is a lightweight framework for efficient Gaussian image representation.<n>We introduce Deep Gaussian Prior, implemented as a conditional network to capture the spatial distribution of Gaussian primitives.<n> Experiments demonstrate that this disentangled architecture achieves high-quality reconstruction in a single forward pass, followed by minimal fine-tuning.
  arXiv  Detail & Related papers  (2025-12-14T17:23:28Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• DINO-Tok: Adapting DINO for Visual Tokenizers [52.194754463297706]
  DINO-Tok is a visual tokenizer that unifies hierarchical representations into an information-complete latent space.<n>On ImageNet, DINO-Tok achieves state-of-the-art reconstruction performance, reaching 28.54 PSNR for autoencoding and 23.98 PSNR for VQ-based modeling.
  arXiv  Detail & Related papers  (2025-11-25T18:00:00Z)
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation [66.73899356886652]
  We build an image tokenizer directly atop pre-trained vision foundation models.<n>Our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality.<n>It further boosts autoregressive (AR) generation -- achieving a gFID of 2.07 on ImageNet benchmarks.
  arXiv  Detail & Related papers  (2025-07-11T09:32:45Z)
• Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesis [57.7367843129838]
  Recent image generation schemes typically capture image distribution in a pre-constructed latent space relying on a frozen image tokenizer.<n>We propose a novel plug-and-play tokenizer training scheme to facilitate latent space construction.
  arXiv  Detail & Related papers  (2025-03-11T12:09:11Z)
• GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting [64.84383010238908]
  We propose an effective image tokenizer with 2D Gaussian Splatting as a solution.<n>In general, our framework integrates the local influence of 2D Gaussian distribution into the discrete space.<n> Competitive reconstruction performances on CIFAR, Mini-Net, and ImageNet-1K demonstrate the effectiveness of our framework.
  arXiv  Detail & Related papers  (2025-01-26T17:56:11Z)
• XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation [54.2574228021317]
  We present XQ-GAN, an image tokenization framework designed for both image reconstruction and generation tasks.<n>Our framework integrates state-of-the-art quantization techniques, including vector quantization (VQ), residual quantization (RQ), multi-scale residual quantization (MSVQ), product quantization (PQ), and binary spherical quantization (BSQ)<n>On the standard ImageNet 256x256 benchmark, our released model achieves an rFID of 0.64, significantly surpassing MAGVIT-v2 (0.9 rFID) and VAR (0.9 rFID)
  arXiv  Detail & Related papers  (2024-12-02T17:58:06Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [52.598772767324036]
  We introduce Image-GS, a content-adaptive image representation based on 2D Gaussians radiance.<n>It supports hardware-friendly rapid access for real-time usage, requiring only 0.3K MACs to decode a pixel.<n>We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation [41.029441562130984]
  Two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images. Our proposed modulated VQGAN is able to greatly improve the reconstructed image quality as well as provide high-fidelity image generation.
  arXiv  Detail & Related papers  (2022-09-19T13:26:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.