VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation

• URL: http://arxiv.org/abs/2505.13439v1
• Date: Mon, 19 May 2025 17:59:01 GMT
• Title: VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation
• Authors: Huawei Lin, Tong Geng, Zhaozhuo Xu, Weijie Zhao,
• Abstract summary: A visual tokenizer (VT) maps continuous pixel inputs to discrete token sequences.<n>Current discrete VTs fall significantly behind continuous variational autoencoders (VAEs), leading to degraded image reconstructions and poor preservation of details and text.<n>Existing benchmarks focus on end-to-end generation quality, without isolating VT performance.<n>We introduce VTBench, a comprehensive benchmark that systematically evaluates VTs across three core tasks: Image Reconstruction, Detail Preservation, and Text Preservation.
• Score: 11.529598741483076
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive (AR) models have recently shown strong performance in image generation, where a critical component is the visual tokenizer (VT) that maps continuous pixel inputs to discrete token sequences. The quality of the VT largely defines the upper bound of AR model performance. However, current discrete VTs fall significantly behind continuous variational autoencoders (VAEs), leading to degraded image reconstructions and poor preservation of details and text. Existing benchmarks focus on end-to-end generation quality, without isolating VT performance. To address this gap, we introduce VTBench, a comprehensive benchmark that systematically evaluates VTs across three core tasks: Image Reconstruction, Detail Preservation, and Text Preservation, and covers a diverse range of evaluation scenarios. We systematically assess state-of-the-art VTs using a set of metrics to evaluate the quality of reconstructed images. Our findings reveal that continuous VAEs produce superior visual representations compared to discrete VTs, particularly in retaining spatial structure and semantic detail. In contrast, the degraded representations produced by discrete VTs often lead to distorted reconstructions, loss of fine-grained textures, and failures in preserving text and object integrity. Furthermore, we conduct experiments on GPT-4o image generation and discuss its potential AR nature, offering new insights into the role of visual tokenization. We release our benchmark and codebase publicly to support further research and call on the community to develop strong, general-purpose open-source VTs.

Related papers

• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• Visual CoT Makes VLMs Smarter but More Fragile [79.32638667101817]
  Chain-of-Thought (CoT) techniques have significantly enhanced reasoning in Vision-Language Models (VLMs)<n>Visual CoT integrates explicit visual edits, such as cropping or annotating regions of interest, into the reasoning process.<n>We present the first systematic evaluation of Visual CoT robustness under visual perturbations.
  arXiv  Detail & Related papers  (2025-09-28T10:19:59Z)
• VIVAT: Virtuous Improving VAE Training through Artifact Mitigation [4.295130967329365]
  This paper introduces VIVAT, a systematic approach to mitigating common artifacts in KL-VAE training without requiring radical architectural changes.<n>We present a detailed taxonomy of five prevalent artifacts - color shift, grid patterns, blur, corner and droplet artifacts - and analyze their root causes.
  arXiv  Detail & Related papers  (2025-06-09T15:27:03Z)
• TokBench: Evaluating Your Visual Tokenizer before Visual Generation [75.38270351179018]
  We analyze text and face reconstruction quality across various scales for different image tokenizers and VAEs.<n>Our results show modern visual tokenizers still struggle to preserve fine-grained features, especially at smaller scales.
  arXiv  Detail & Related papers  (2025-05-23T17:52:16Z)
• Extrapolated Urban View Synthesis Benchmark [53.657271730352214]
  Photo simulators are essential for the training and evaluation of vision-centric autonomous vehicles (AVs)<n>At their core is Novel View Synthesis (NVS), a capability that generates diverse unseen viewpoints to accommodate the broad and continuous pose distribution of AVs.<n>Recent advances in radiance fields, such as 3D Gaussian Splatting, achieve photorealistic rendering at real-time speeds and have been widely used in modeling large-scale driving scenes.<n>We will release the data to help advance self-driving and urban robotics simulation technology.
  arXiv  Detail & Related papers  (2024-12-06T18:41:39Z)
• Assessing Brittleness of Image-Text Retrieval Benchmarks from Vision-Language Models Perspective [44.045767657945895]
  We examine the brittleness of the image-text retrieval (ITR) evaluation pipeline with a focus on concept granularity. We evaluate four diverse state-of-the-art Vision-Language models on both the standard and fine-grained datasets under zero-shot conditions. The results demonstrate that although perturbations generally degrade model performance, the fine-grained datasets exhibit a smaller performance drop than their standard counterparts.
  arXiv  Detail & Related papers  (2024-07-21T18:08:44Z)
• SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration [20.873676111265656]
  SSP-IR aims to fully exploit semantic and structure priors from low-quality images.<n>Our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-07-04T04:55:14Z)
• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving [55.93813178692077]
  We present RoboBEV, an extensive benchmark suite designed to evaluate the resilience of BEV algorithms.<n>We assess 33 state-of-the-art BEV-based perception models spanning tasks like detection, map segmentation, depth estimation, and occupancy prediction.<n>Our experimental results also underline the efficacy of strategies like pre-training and depth-free BEV transformations in enhancing robustness against out-of-distribution data.
  arXiv  Detail & Related papers  (2024-05-27T17:59:39Z)
• Denoising Vision Transformers [43.03068202384091]
  We propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT) In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision.
  arXiv  Detail & Related papers  (2024-01-05T18:59:52Z)
• Generative Negative Text Replay for Continual Vision-Language Pretraining [95.2784858069843]
  Vision-language pre-training has attracted increasing attention recently. Massive data are usually collected in a streaming fashion. We propose a multi-modal knowledge distillation between images and texts to align the instance-wise prediction between old and new models.
  arXiv  Detail & Related papers  (2022-10-31T13:42:21Z)
• A Comprehensive Study of Vision Transformers on Dense Prediction Tasks [10.013443811899466]
  Convolutional Neural Networks (CNNs) have been the standard choice in vision tasks. Recent studies have shown that Vision Transformers (VTs) achieve comparable performance in challenging tasks such as object detection and semantic segmentation. This poses several questions about their generalizability, robustness, reliability, and texture bias when used to extract features for complex tasks.
  arXiv  Detail & Related papers  (2022-01-21T13:18:16Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.