Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse

• URL: http://arxiv.org/abs/2605.06870v2
• Date: Tue, 12 May 2026 01:29:33 GMT
• Title: Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse
• Abstract summary: We show theoretically and empirically that dimension collapse causes a hard loss lower bound that various codebook improvement techniques fail to surpass.<n>We propose a "warm-up phase" that trains the model as an (unquantized) autoencoder before introducing VQ.<n>On both synthetic experiments and large-scale image (VQGAN) and audio (WavTokenizer) VQ-VAEs, we show that AE Warm-Up successfully restores representation dimension.
• Score: 63.31488859236551
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While many approaches to improve VQ-VAE performance focus on codebook size and utilization, the effect of dimensional collapse, where trained VQ-VAE representations live in an extremely low-dimensional subspace (1-2% of full rank), remains unaddressed. We show theoretically and empirically that dimension collapse causes a hard loss lower bound that various codebook improvement techniques fail to surpass. Our analytic framework extends the sequential learning effect of Saxe et al. [2014] by introducing ideas from rate-distortion theory and explains how the latent collapse is caused by the VQ suppressing lower-variance directions. Our theory justifies a simple solution: a "warm-up phase" that trains the model as an (unquantized) autoencoder before introducing VQ. On both synthetic experiments and large-scale image (VQGAN) and audio (WavTokenizer) VQ-VAEs, we show that AE Warm-Up successfully restores representation dimension, leading to lower reconstruction and perceptual loss at the same training budget. Across codebook sizes $K \in$ {$2^{10}, 2^{14}, 2^{16}$}, AE warm-up raises VQGAN codebook effective dimension from 3-5 to 17-19 and reduces rFID by 17-35%; on WavTokenizer at $K \in$ {$2^{13}, 2^{14}$}, it raises codebook dimension from 4 to 17-19 and improves PESQ by 11-14%. We empirically characterize how warm-up duration governs the achievable final loss. In agreement with experiment, our theoretical analysis predicts downstream performance as a function of warm-up length, enabling an adaptive criterion for switching from AE Warm-up to VQ-VAE training.

Related papers

• Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression [53.041405703471895]
  We propose RDVQ, a unified framework that enables end-to-end RD optimization for low-bitrate image compression.<n>We develop an autoregressive entropy model that supports accurate entropy modeling and test-time rate control.<n>Compared with RD2K-val, RDVQ reduces by up to 75.71% on DISTS and 37.63% on LPIPS on DIV2K-val.
  arXiv  Detail & Related papers  (2026-04-12T09:25:24Z)
• Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization [17.805458929887894]
  We propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training.<n>We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling.<n>ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.
  arXiv  Detail & Related papers  (2026-03-17T18:03:09Z)
• Sat-EnQ: Satisficing Ensembles of Weak Q-Learners for Reliable and Compute-Efficient Reinforcement Learning [0.0]
  We introduce Sat-EnQ, a framework that learns to be good enough'' before optimizing aggressively.<n>In Phase 1, we train an ensemble of lightweight Q-networks under a satisficing objective that limits early value growth.<n>In Phase 2, the ensemble is distilled into a larger network and fine-tuned with standard Double DQN.
  arXiv  Detail & Related papers  (2025-12-28T12:41:09Z)
• Quantize-then-Rectify: Efficient VQ-VAE Training [71.92014859992263]
  This work demonstrates that a pre-trained VAE can be efficiently transformed into a VQ-VAE by quantization noise within the VAE's tolerance threshold.<n>We present textbfQuantize-then-Rectify (ReVQ), a framework leveraging pre-trained VAEs to enable rapid VQ-VAE training with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-07-14T17:59:41Z)
• Scalable Image Tokenization with Index Backpropagation Quantization [74.15447383432262]
  Index Backpropagation Quantization (IBQ) is a new VQ method for the joint optimization of all codebook embeddings and the visual encoder.<n>IBQ enables scalable training of visual tokenizers and, for the first time, achieves a large-scale codebook with high dimension ($256$) and high utilization.
  arXiv  Detail & Related papers  (2024-12-03T18:59:10Z)
• Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis [30.654501418221475]
  We show that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation ability of generative transformers. We propose Semantic-Quantized GAN (SeQ-GAN) with two learning phases to balance the two objectives. Our SeQ-GAN (364M) achieves Frechet Inception Distance (FID) of 6.25 and Inception Score (IS) of 140.9 on 256x256 ImageNet generation.
  arXiv  Detail & Related papers  (2022-12-06T17:58:38Z)
• EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback [65.60461294185853]
  We propose a new error feedback mechanism, EF21, based on the construction of a Markov compressor induced by a contractive compressor.<n>Several of these techniques have not been previously analyzed in combination with EF21.
  arXiv  Detail & Related papers  (2021-10-07T09:29:14Z)
• Contrast and Classify: Training Robust VQA Models [60.80627814762071]
  We propose a novel training paradigm (ConClaT) that optimize both cross-entropy and contrastive losses. We find that optimizing both losses -- either alternately or jointly -- is key to effective training.
  arXiv  Detail & Related papers  (2020-10-13T00:23:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.