VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

• URL: http://arxiv.org/abs/2607.19575v1
• Date: Tue, 21 Jul 2026 21:01:49 GMT
• Title: VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers
• Abstract summary: VQ-Transplant is a framework that enables integration of new VQ modules into frozen, pre-trained tokenizers.<n>The proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining.<n>We find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models.
• Score: 17.068345957829635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose {\bf VQ-Transplant}, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95\%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.

Related papers

• Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression [53.041405703471895]
  We propose RDVQ, a unified framework that enables end-to-end RD optimization for low-bitrate image compression.<n>We develop an autoregressive entropy model that supports accurate entropy modeling and test-time rate control.<n>Compared with RD2K-val, RDVQ reduces by up to 75.71% on DISTS and 37.63% on LPIPS on DIV2K-val.
  arXiv  Detail & Related papers  (2026-04-12T09:25:24Z)
• Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization [17.805458929887894]
  We propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training.<n>We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling.<n>ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.
  arXiv  Detail & Related papers  (2026-03-17T18:03:09Z)
• Is Hierarchical Quantization Essential for Optimal Reconstruction? [0.0]
  We show that inadequate codebook utilization limits single-level VQ-VAEs and that overly high-dimensional embeddings destabilize quantization and increase codebook collapse.<n>Our results demonstrate that when representational budgets are matched, and codebook collapse is mitigated, single-level VQ-VAEs can match the reconstruction fidelity of hierarchical variants.
  arXiv  Detail & Related papers  (2026-01-29T19:09:50Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• Quantize-then-Rectify: Efficient VQ-VAE Training [71.92014859992263]
  This work demonstrates that a pre-trained VAE can be efficiently transformed into a VQ-VAE by quantization noise within the VAE's tolerance threshold.<n>We present textbfQuantize-then-Rectify (ReVQ), a framework leveraging pre-trained VAEs to enable rapid VQ-VAE training with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-07-14T17:59:41Z)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation [55.12070409045766]
  Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years.<n>Current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-13T07:57:38Z)
• Preventing Local Pitfalls in Vector Quantization via Optimal Transport [77.15924044466976]
  We introduce OptVQ, a novel vector quantization method that employs the Sinkhorn algorithm to optimize the optimal transport problem.<n>Our experiments on image reconstruction tasks demonstrate that OptVQ achieves 100% codebook utilization and surpasses current state-of-the-art VQNs in reconstruction quality.
  arXiv  Detail & Related papers  (2024-12-19T18:58:14Z)
• XQ-GAN: An Open-source Image Tokenization Framework for Autoregressive Generation [54.2574228021317]
  We present XQ-GAN, an image tokenization framework designed for both image reconstruction and generation tasks.<n>Our framework integrates state-of-the-art quantization techniques, including vector quantization (VQ), residual quantization (RQ), multi-scale residual quantization (MSVQ), product quantization (PQ), and binary spherical quantization (BSQ)<n>On the standard ImageNet 256x256 benchmark, our released model achieves an rFID of 0.64, significantly surpassing MAGVIT-v2 (0.9 rFID) and VAR (0.9 rFID)
  arXiv  Detail & Related papers  (2024-12-02T17:58:06Z)
• Few-Shot Image Quality Assessment via Adaptation of Vision-Language Models [93.91086467402323]
  Gradient-Regulated Meta-Prompt IQA Framework (GRMP-IQA) designed to efficiently adapt the visual-language pre-trained model, CLIP, to IQA tasks.<n> GRMP-IQA consists of two core modules: (i) Meta-Prompt Pre-training Module and (ii) Quality-Aware Gradient Regularization.
  arXiv  Detail & Related papers  (2024-09-09T07:26:21Z)
• Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Models [3.7906296809297393]
  We introduce Rate-Adaptive Quantization (RAQ), a multi-rate codebook adaptation framework for VQ-based generative models.<n>RAQ applies a data-driven approach to generate variable-rate codebooks from a single baseline VQ model.<n>Our experiments show that RAQ performs effectively across multiple rates, often outperforming conventional fixed-rate VQ baselines.
  arXiv  Detail & Related papers  (2024-05-23T06:32:42Z)
• Pre-training Tensor-Train Networks Facilitates Machine Learning with Variational Quantum Circuits [70.97518416003358]
  Variational quantum circuits (VQCs) hold promise for quantum machine learning on noisy intermediate-scale quantum (NISQ) devices. While tensor-train networks (TTNs) can enhance VQC representation and generalization, the resulting hybrid model, TTN-VQC, faces optimization challenges due to the Polyak-Lojasiewicz (PL) condition. To mitigate this challenge, we introduce Pre+TTN-VQC, a pre-trained TTN model combined with a VQC.
  arXiv  Detail & Related papers  (2023-05-18T03:08:18Z)
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction [29.040991149922615]
  We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ) We propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. For the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 comparable with QAT and enjoy 240 times faster production of quantized models.
  arXiv  Detail & Related papers  (2021-02-10T13:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.