VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits

• URL: http://arxiv.org/abs/2505.10202v1
• Date: Thu, 15 May 2025 11:58:04 GMT
• Title: VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits
• Authors: Jintian Shao, Hongyi Huang, Jiayi Wu, YiMing Cheng, ZhiYu Wu, You Shan, MingKai Zheng,
• Abstract summary: VQ-Logits is a novel approach to drastically reduce the parameter count and computational load of the Large Language Models output layer.<n>We show that VQ-Logits can achieve up to 99% parameter reduction in the output layer and 6x speedup in logit computation, with only a marginal 4% increase in perplexity compared to full softmax baselines.
• Score: 9.470124763460904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their extensive output vocabularies. The final linear projection layer, mapping hidden states to vocabulary-sized logits, often constitutes a substantial portion of the model's parameters and computational cost during inference. Existing methods like adaptive softmax or hierarchical softmax introduce structural complexities. In this paper, we propose VQ-Logits, a novel approach that leverages Vector Quantization (VQ) to drastically reduce the parameter count and computational load of the LLM output layer. VQ-Logits replaces the large V * dmodel output embedding matrix with a small, shared codebook of K embedding vectors (K << V ). Each token in the vocabulary is mapped to one of these K codebook vectors. The LLM predicts logits over this compact codebook, which are then efficiently "scattered" to the full vocabulary space using the learned or preassigned mapping. We demonstrate through extensive experiments on standard language modeling benchmarks (e.g., WikiText-103, C4) that VQ-Logits can achieve up to 99% parameter reduction in the output layer and 6x speedup in logit computation, with only a marginal 4% increase in perplexity compared to full softmax baselines. We further provide detailed ablation studies on codebook size, initialization, and learning strategies, showcasing the robustness and effectiveness of our approach.

Related papers

• SpeLLM: Character-Level Multi-Head Decoding [6.474939955469084]
  SpeLLM is a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads.<n>In SpeLLM, each of the $k$ linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space.<n>Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1%.
  arXiv  Detail & Related papers  (2025-07-22T08:07:06Z)
• Efficient Whole Slide Pathology VQA via Token Compression [10.122347041204629]
  Whole-slide images (WSIs) in pathology can reach up to 10,000 x 10,000 pixels, posing significant challenges for large language model (MLLM)<n>We propose Token Compression Pathology LLaVA ( TCP-LLaVA), the first MLLM architecture to perform WSI VQA via token compression.
  arXiv  Detail & Related papers  (2025-07-19T06:04:25Z)
• Quantizing Large Language Models for Code Generation: A Differentiated Replication [51.85505914274633]
  Large Language Models (LLMs) have shown an impressive capability in code generation and, specifically, to automatically implement requirements described in natural language.<n>LLMs pose significant challenges related to their memory (and, consequently, carbon) footprint.<n>New frontier for LLM quantization is 4-bit precision, resulting in an average memory footprint reduction of 70%.
  arXiv  Detail & Related papers  (2025-03-10T09:26:08Z)
• QLESS: A Quantized Approach for Data Valuation and Selection in Large Language Model Fine-Tuning [12.371302966300128]
  We propose textbfQLESS (Quantized Low-rank Gradient Similarity Search)<n>QLESS integrates gradient quantization with the LESS framework to enable memory-efficient data valuation and selection.<n> Experiments show QLESS achieves comparable data selection performance to LESS while reducing memory usage by up to 16x.
  arXiv  Detail & Related papers  (2025-02-03T10:52:32Z)
• Addressing Representation Collapse in Vector Quantized Models with One Linear Layer [10.532262196027752]
  Vector Quantization (VQ) is a widely used method for converting continuous representations into discrete codes. VQ models are often hindered by the problem of representation collapse in the latent space. We propose textbfSimVQ, a novel method which re parameterizes the code vectors through a linear transformation layer based on a learnable latent basis.
  arXiv  Detail & Related papers  (2024-11-04T12:40:18Z)
• Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models [6.646510073473929]
  We propose SlimCode, a model-agnostic code simplification solution for Large Language Models. SlimCode can improve the state-of-the-art technique by 9.46% and 5.15% in terms of MRR and BLEU score on code search and summarization.
  arXiv  Detail & Related papers  (2024-05-18T06:15:52Z)
• Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs [61.40047491337793]
  We present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations of large language models. HomeR uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. A token reduction technique precedes each merging, ensuring memory usage efficiency.
  arXiv  Detail & Related papers  (2024-04-16T06:34:08Z)
• FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping [49.66872823080736]
  Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation. To mitigate overload incurred during generation, several early-exit and layer-dropping strategies have been proposed. We propose FFN-SkipLLM, which is an input-adaptive feed-forward skipping strategy.
  arXiv  Detail & Related papers  (2024-04-05T02:35:43Z)
• SparseCoder: Identifier-Aware Sparse Transformer for File-Level Code Summarization [51.67317895094664]
  This paper studies file-level code summarization, which can assist programmers in understanding and maintaining large source code projects. We propose SparseCoder, an identifier-aware sparse transformer for effectively handling long code sequences.
  arXiv  Detail & Related papers  (2024-01-26T09:23:27Z)
• Extreme Compression of Large Language Models via Additive Quantization [59.3122859349777]
  Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval. We provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed.
  arXiv  Detail & Related papers  (2024-01-11T18:54:44Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models [22.06402870816756]
  Large language models (LLMs) have been applied in various applications due to their astonishing capabilities. This paper presents LLMLingua, a coarse-to-fine prompt compression method that involves a budget controller to maintain semantic integrity. We show that the proposed approach yields state-of-the-art performance and allows for up to 20x compression with little performance loss.
  arXiv  Detail & Related papers  (2023-10-09T14:10:21Z)
• Soft Convex Quantization: Revisiting Vector Quantization with Convex Optimization [40.1651740183975]
  We propose Soft Convex Quantization (SCQ) as a direct substitute for Vector Quantization (VQ) SCQ works like a differentiable convex optimization (DCO) layer. We demonstrate its efficacy on the CIFAR-10, GTSRB and LSUN datasets.
  arXiv  Detail & Related papers  (2023-10-04T17:45:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.