Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization

• URL: http://arxiv.org/abs/2603.17052v1
• Date: Tue, 17 Mar 2026 18:32:21 GMT
• Title: Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
• Abstract summary: Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors.<n>Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored.<n>This study systematically investigates the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings.
• Score: 15.662726701547896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data representations for large language models, diffusion models, and other generative models. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored. In this study, we systematically investigate the issue of collapses in vector quantization, where collapsed representations are observed across discrete codebook tokens and continuous latent embeddings. By leveraging both synthetic and real datasets, we identify the severity of each type of collapses and triggering conditions. Our analysis reveals that random initialization and limited encoder capacity result in tokens collapse and embeddings collapse. Building on these findings, we propose potential solutions aimed at mitigating each collapse. To the best of our knowledge, this is the first comprehensive study examining representation collapsing problems in vector quantization.

Related papers

• Why shared attention vectors fail: a case for outcome-indexed tuning [0.0]
  We show that under multi-outcome learning, where models predict more than one outcome, a globally shared attentional vector becomes unstable.<n>We introduce an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation.
  arXiv  Detail & Related papers  (2026-09-08T11:52:08Z)
• To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoning [34.300040798165774]
  We investigate the ability of decoder-only transformer models to perform abstract symbolic reasoning.<n>We show that a particular representational collapse also has a crucial role: the unembeddings of unseen tokens collapse to nearly the same vector during training.
  arXiv  Detail & Related papers  (2026-04-23T12:51:10Z)
• SIGMA: Scalable Spectral Insights for LLM Collapse [51.863164847253366]
  We introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework for model collapse.<n>By utilizing benchmarks that deriving and deterministic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space.<n>We demonstrate that SIGMA effectively captures the transition towards states, offering both theoretical insights into the mechanics of collapse.
  arXiv  Detail & Related papers  (2026-01-06T19:47:11Z)
• Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study [19.74160064041426]
  Two critical issues in vector quantization methods are training instability and codebook collapse.<n>We employ the Wasserstein distance to align these two distributions, achieving near 100% codebook utilization.<n>Both empirical and theoretical analyses validate the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2025-06-18T02:43:40Z)
• Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation [85.82112629564942]
  We propose TokenBridge, which maintains the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens.<n>We introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism.<n>Our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction.
  arXiv  Detail & Related papers  (2025-03-20T17:59:59Z)
• Representation Collapsing Problems in Vector Quantization [2.2750239768387255]
  Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. Despite its prevalence, the characteristics and behaviors of vector quantization in generative models remain largely underexplored.
  arXiv  Detail & Related papers  (2024-11-25T16:32:29Z)
• Quantization of Large Language Models with an Overdetermined Basis [73.79368761182998]
  We introduce an algorithm for data quantization based on the principles of Kashin representation. Our findings demonstrate that Kashin Quantization achieves competitive or superior quality in model performance.
  arXiv  Detail & Related papers  (2024-04-15T12:38:46Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• Attribute Graphs Underlying Molecular Generative Models: Path to Learning with Limited Data [42.517927809224275]
  We provide an algorithm that relies on perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover an attribute graph. We show that one can fit an effective graphical model that models a structural equation model between latent codes. Using a pre-trained generative autoencoder trained on a large dataset of small molecules, we demonstrate that the graphical model can be used to predict a specific property.
  arXiv  Detail & Related papers  (2022-07-14T19:20:30Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.