Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantization

• URL: http://arxiv.org/abs/2602.01140v1
• Date: Sun, 01 Feb 2026 10:22:35 GMT
• Title: Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantization
• Authors: Haochen You, Heng Zhang, Hongyang He, Yuqi Li, Baojing Liu,
• Abstract summary: We introduce a unified surrogate framework that keeps hard assignments in the forward pass while making VQ fully differentiable.<n>GRIT-VQ consistently improves reconstruction error, generative quality, and accuracy compared to existing VQ variants.
• Score: 11.898954874548073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vector quantization (VQ) underpins modern generative and representation models by turning continuous latents into discrete tokens. Yet hard nearest-neighbor assignments are non-differentiable and are typically optimized with heuristic straight-through estimators, which couple the update step size to the quantization gap and train each code in isolation, leading to unstable gradients and severe codebook under-utilization at scale. In this paper, we introduce GRIT-VQ (Generalized Radius and Integrated Transform-Vector Quantization), a unified surrogate framework that keeps hard assignments in the forward pass while making VQ fully differentiable. GRIT-VQ replaces the straight-through estimator with a radius-based update that moves latents along the quantization direction with a controllable, geometry-aware step, and applies a data-agnostic integrated transform to the codebook so that all codes are updated through shared parameters instead of independently. Our theoretical analysis clarifies the fundamental optimization dynamics introduced by GRIT-VQ, establishing conditions for stable gradient flow, coordinated codebook evolution, and reliable avoidance of collapse across a broad family of quantizers. Across image reconstruction, image generation, and recommendation tokenization benchmarks, GRIT-VQ consistently improves reconstruction error, generative quality, and recommendation accuracy while substantially increasing codebook utilization compared to existing VQ variants.

Related papers

• Beyond Stationarity: Rethinking Codebook Collapse in Vector Quantization [12.305907179979426]
  We show that as the encoder drifts, unselected code vectors fail to receive updates and gradually become inactive.<n>To address this, we propose two new methods: Non-Stationary Vector Quantization (NSVQ) and Transformer-based Vector Quantization (TransVQ)<n> Experiments on the CelebA-HQ dataset demonstrate that both methods achieve near-complete codebook utilization and superior reconstruction quality.
  arXiv  Detail & Related papers  (2026-02-21T16:36:50Z)
• Is Hierarchical Quantization Essential for Optimal Reconstruction? [0.0]
  We show that inadequate codebook utilization limits single-level VQ-VAEs and that overly high-dimensional embeddings destabilize quantization and increase codebook collapse.<n>Our results demonstrate that when representational budgets are matched, and codebook collapse is mitigated, single-level VQ-VAEs can match the reconstruction fidelity of hierarchical variants.
  arXiv  Detail & Related papers  (2026-01-29T19:09:50Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling [22.005420177236804]
  We propose VAEVQ, which comprises three key components: (1) Variational Latent Quantization (VLQ), replacing the AE with a VAE for quantization to leverage its structured and smooth latent space, thereby facilitating more effective codeword activation; (2) Representation Coherence Strategy (RCS), adaptively modulating the alignment strength between pre- and post-quantization features to enhance consistency and prevent overfitting to noise; and (3) Distribution Consistency Regularization (DCR), aligning the entire codebook distribution with the continuous latent distribution to improve utilization.
  arXiv  Detail & Related papers  (2025-11-10T09:07:23Z)
• Scalable Training for Vector-Quantized Networks with 100% Codebook Utilization [60.294965457786844]
  Vector quantization (VQ) is a key component in discrete tokenizers for image generation.<n>VQBridge is a robust, scalable, and efficient projector based on the map function method.<n>FVQ attains 100% codebook usage even with a 262k-codebook.
  arXiv  Detail & Related papers  (2025-09-12T11:08:21Z)
• XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation [54.2574228021317]
  We present XQ-GAN, an image tokenization framework designed for both image reconstruction and generation tasks.<n>Our framework integrates state-of-the-art quantization techniques, including vector quantization (VQ), residual quantization (RQ), multi-scale residual quantization (MSVQ), product quantization (PQ), and binary spherical quantization (BSQ)<n>On the standard ImageNet 256x256 benchmark, our released model achieves an rFID of 0.64, significantly surpassing MAGVIT-v2 (0.9 rFID) and VAR (0.9 rFID)
  arXiv  Detail & Related papers  (2024-12-02T17:58:06Z)
• Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior [5.862123282894087]
  We introduce the Vector Quantized Variational Autoencoder (VQ-VAE) VQ-VAE is a type of variational autoencoder using discrete embedding as latent. We show that GM-VQ improves codebook utilization and reduces information loss without relying on handcrafteds.
  arXiv  Detail & Related papers  (2024-10-14T05:58:11Z)
• HyperVQ: MLR-based Vector Quantization in Hyperbolic Space [56.4245885674567]
  A common solution is to employ Vector Quantization (VQ) within VQ Variational Autoencoders (VQVAEs)<n>We introduce HyperVQ, a novel approach that formulates VQ as a hyperbolic Multinomial Logistic Regression (MLR) problem.<n>Our experiments demonstrate that HyperVQ matches traditional VQ in generative and reconstruction tasks, while surpassing it in discriminative performance.
  arXiv  Detail & Related papers  (2024-03-18T03:17:08Z)
• HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes [18.57499609338579]
  We propose a novel framework to learn hierarchical discrete representation on the basis of the variational Bayes framework, called hierarchically quantized variational autoencoder (HQ-VAE) HQ-VAE naturally generalizes the hierarchical variants of VQ-VAE, such as VQ-VAE-2 and residual-quantized VAE (RQ-VAE) Our comprehensive experiments on image datasets show that HQ-VAE enhances codebook usage and improves reconstruction performance.
  arXiv  Detail & Related papers  (2023-12-31T01:39:38Z)
• Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization [40.1651740183975]
  We propose Soft Convex Quantization (SCQ) as a direct substitute for Vector Quantization (VQ) SCQ works like a differentiable convex optimization (DCO) layer. We demonstrate its efficacy on the CIFAR-10, GTSRB and LSUN datasets.
  arXiv  Detail & Related papers  (2023-10-04T17:45:14Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation [85.5379146125199]
  Variational Auto-Encoder (VAE) has been widely adopted in text generation. We propose TRACE, a Transformer-based recurrent VAE structure.
  arXiv  Detail & Related papers  (2022-10-22T10:25:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.