Vision Foundation Models as Generalist Tokenizers for Image Generation

• URL: http://arxiv.org/abs/2605.18390v1
• Date: Mon, 18 May 2026 13:38:43 GMT
• Title: Vision Foundation Models as Generalist Tokenizers for Image Generation
• Abstract summary: We build a generalist image tokenizer on top of a frozen vision foundation model (VFM)<n>We propose VFMTok, a tokenizer capable of operating seamlessly in both discrete and continuous latent spaces.<n>We find that the specific self-supervised learning objectives utilized during VFM pre-training dictate its effectiveness as a tokenizer.
• Score: 43.17659097958283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we explore the largely unexplored direction of building a generalist image tokenizer directly on top of a frozen vision foundation model (VFM). To build this tokenizer, we utilize a frozen VFM as the encoder and introduce two key innovations: (1) a region-adaptive quantization framework to eliminate spatial redundancy in standard 2D grid features, and (2) a semantic reconstruction objective that aligns the decoded outputs with the VFM's representations to preserve semantic fidelity. Grounded in these designs, we propose VFMTok, a generalist visual tokenizer capable of operating seamlessly in both discrete and continuous latent spaces. VFMTok achieves substantial improvements in synthesis quality while drastically enhancing token efficiency. For discrete autoregressive (AR) generation, it accelerates model convergence by \textbf{3 times} and achieves a state-of-the-art gFID of \textbf{1.36} on ImageNet class-conditional synthesis. Similarly, for continuous-space generation, integrating VFMTok with a denoising model yields an exceptional gFID of \textbf{1.25}. Furthermore, because the latent space inherently captures rich spatial semantics, VFMTok enables high-fidelity class-conditional synthesis without classifier-free guidance (\textbf{w/o CFG}) across both generative paradigms, significantly accelerating inference speed. Beyond these remarkable empirical results, we systematically investigate the underlying mechanisms of our approach. We discover that the specific self-supervised learning objectives utilized during VFM pre-training dictate its effectiveness as a tokenizer. Specifically, a VFM jointly optimized with global contrastive learning and latent masked image modeling provides the optimal representations for image tokenization. These insights establish a strong foundation and offer valuable guidance for the design of future image tokenizers.

Related papers

• Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis [93.7120829382642]
  Nemotron-Labs-Diffusion-Image is a state-of-the-art masked discrete diffusion model for text-to-image synthesis.<n>Standard MDMs lack self-correcting capability because discrete tokens cannot be modified once they are unmasked.<n>Nemotron-Labs-Diffusion-Image incorporates a token-editing mechanism that enables the model to dynamically revise already-unmasked tokens during inference.
  arXiv  Detail & Related papers  (2026-06-29T05:48:41Z)
• IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder [74.25043153401586]
  We propose Ideal, an In-depth Alignment framework for discrete representation autoencoding.<n>By aligning quantized tokens with both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve both visual fidelity and rich semantics.
  arXiv  Detail & Related papers  (2026-06-09T16:53:30Z)
• DINO-SAE: DINO Spherical Autoencoder for High-Fidelity Image Reconstruction and Generation [47.409626500688866]
  We present the DINO Spherical Autoencoder (DINO-SAE), a framework that bridges semantic representation and pixel-level reconstruction.<n>Our approach achieves state-of-the-art reconstruction quality, reaching 0.37 rFID and 26.2 dB PSNR, while maintaining strong semantic alignment to the pretrained VFM.
  arXiv  Detail & Related papers  (2026-01-30T12:25:34Z)
• DreamVAR: Taming Reinforced Visual Autoregressive Model for High-Fidelity Subject-Driven Image Generation [108.71044040025374]
  We present a novel framework for subject-driven image synthesis built upon a Visual Autoregressive model that employs next-scale prediction.<n>We show that Dreamthe achieves superior appearance preservation compared to leading diffusion-based methods.
  arXiv  Detail & Related papers  (2026-01-30T03:32:29Z)
• REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion [11.138412313646995]
  We introduce REGLUE, a unified latent diffusion framework.<n>A lightweight convolutional semantic nonlinearly aggregates multi-layer VFM features into a low-dimensional, spatially structured representation.<n>On ImageNet 256x256, REGLUE consistently improves FID and convergence over SiT-B/2 and SiT-XL/2 baselines, as well as over REPA, ReDi, and REG.
  arXiv  Detail & Related papers  (2025-12-18T15:10:42Z)
• Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models [45.63522160275318]
  Vision Foundation Model Variational Autoencoder (VFM-VAE) designed to resolve inherent tension between VFM's semantic focus and need for pixel-level fidelity.<n>Our system reaches a gFID (w/o CFG) of 2.20 in merely 80 epochs (a 10x speedup over prior tokenizers)
  arXiv  Detail & Related papers  (2025-10-21T09:30:45Z)
• IAR2: Improving Autoregressive Visual Generation with Semantic-Detail Associated Token Prediction [77.06211178777939]
  IAR2 is an advanced autoregressive framework that enables a hierarchical semantic-detail synthesis process.<n>We show that IAR2 sets a new state-of-the-art for autoregressive image generation, achieving a FID of 1.50 on ImageNet.
  arXiv  Detail & Related papers  (2025-10-08T12:08:21Z)
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation [52.261584726401686]
  We present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model.<n>Based on these designs, our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality.
  arXiv  Detail & Related papers  (2025-07-11T09:32:45Z)
• UGMAE: A Unified Framework for Graph Masked Autoencoders [67.75493040186859]
  We propose UGMAE, a unified framework for graph masked autoencoders. We first develop an adaptive feature mask generator to account for the unique significance of nodes. We then design a ranking-based structure reconstruction objective joint with feature reconstruction to capture holistic graph information.
  arXiv  Detail & Related papers  (2024-02-12T19:39:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.