Channel-wise Vector Quantization

• URL: http://arxiv.org/abs/2605.26089v2
• Date: Mon, 01 Jun 2026 17:08:42 GMT
• Title: Channel-wise Vector Quantization
• Abstract summary: We present Channel-wise Vector Quantization (CVQ), a novel image tokenization paradigm that replaces patch-wise tokens with channel-wise tokens.<n>CVQ represents an image as discrete levels of visual details, rather than as a grid of patches.<n>We show that CVQ achieves 100% codebook utilization with a 16K+ codebook size without any bells and whistles, and substantially improves reconstruction quality over conventional vector quantization.
• Score: 59.96775089792198
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Channel-wise Vector Quantization (CVQ), a novel image tokenization paradigm that replaces patch-wise tokens with channel-wise tokens. Unlike conventional vector quantization, which assigns a discrete token to each patch feature vector, CVQ quantizes each channel of the feature map. This formulation represents an image as discrete levels of visual details, rather than as a grid of spatial patches. Based on CVQ, we introduce a new visual autoregressive framework with "next-channel prediction". Instead of rendering images patch by patch in raster order, our Channel-wise Autoregressive (CAR) model predicts image channels sequentially, producing progressively enriched visual details. Specifically, it first sketches global structure and then refines fine-grained attributes, akin to a human artist's workflow. Empirically, we show that: (1) CVQ achieves 100% codebook utilization with a 16K+ codebook size without any bells and whistles, and substantially improves reconstruction quality over conventional VQ; and (2) CAR attains a DPG score of 86.7 and a GenEval score of 0.79, demonstrating strong effectiveness for text-to-image generation.

Related papers

• NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale [101.57871281101747]
  NextStep-1 achieves state-of-the-art performance for autoregressive models in text-to-image generation tasks.<n>Our method shows strong performance in image editing, highlighting the power and versatility of our unified approach.
  arXiv  Detail & Related papers  (2025-08-14T14:54:22Z)
• Hita: Holistic Tokenizer for Autoregressive Image Generation [56.81871174745175]
  We introduce textitHita, a novel image tokenizer for autoregressive (AR) image generation.<n>It introduces a holistic-to-local tokenization scheme with learnable holistic queries and local patch tokens.
  arXiv  Detail & Related papers  (2025-07-03T06:44:26Z)
• Highly Compressed Tokenizer Can Generate Without Training [0.5033155053523042]
  1D image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens.<n>We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities.<n>Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.
  arXiv  Detail & Related papers  (2025-06-09T21:45:03Z)
• Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model [69.79418000132995]
  AliTok is a novel Aligned Tokenizer that alters the dependency structure of the token sequence.<n>Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on the ImageNet-256 benchmark.
  arXiv  Detail & Related papers  (2025-06-05T17:45:10Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization [43.12251414524675]
  Masked Image Modeling (MIM) with Vector Quantization (VQ) has achieved great success in both self-supervised pre-training and image generation.<n>We propose MergeVQ, which incorporates token merging techniques into VQ-based generative models to bridge the gap between image generation and visual representation learning.
  arXiv  Detail & Related papers  (2025-04-01T17:39:19Z)
• Improving Autoregressive Image Generation through Coarse-to-Fine Token Prediction [4.900334213807624]
  We show how to enjoy the benefits of large codebooks without making autoregressive modeling more difficult.<n>Our framework consists of two stages: (1) an autoregressive model that sequentially predicts coarse labels for each token in the sequence, and (2) an auxiliary model that simultaneously predicts fine-grained labels for all tokens conditioned on their coarse labels.
  arXiv  Detail & Related papers  (2025-03-20T14:41:29Z)
• Autoregressive Image Generation with Randomized Parallel Decoding [28.352741116124538]
  We introduce ARPG, a novel visual autoregressive model that enables randomized parallel generation.<n>ARPG achieves over a 30 times speedup in inference and a 75 percent reduction in memory consumption.<n>On the ImageNet-1K 256 benchmark, our approach attains an FID of 1.83 with only 32 sampling steps.
  arXiv  Detail & Related papers  (2025-03-13T17:19:51Z)
• Scalable Image Tokenization with Index Backpropagation Quantization [74.15447383432262]
  Index Backpropagation Quantization (IBQ) is a new VQ method for the joint optimization of all codebook embeddings and the visual encoder.<n>IBQ enables scalable training of visual tokenizers and, for the first time, achieves a large-scale codebook with high dimension ($256$) and high utilization.
  arXiv  Detail & Related papers  (2024-12-03T18:59:10Z)
• Transformer-based Image Generation from Scene Graphs [11.443097632746763]
  Graph-structured scene descriptions can be efficiently used in generative models to control the composition of the generated image. Previous approaches are based on the combination of graph convolutional networks and adversarial methods for layout prediction and image generation. We show how employing multi-head attention to encode the graph information can improve the quality of the sampled data.
  arXiv  Detail & Related papers  (2023-03-08T14:54:51Z)
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation [95.02406834386814]
  Parti treats text-to-image generation as a sequence-to-sequence modeling problem. Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. PartiPrompts (P2) is a new holistic benchmark of over 1600 English prompts.
  arXiv  Detail & Related papers  (2022-06-22T01:11:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.