Does quantization affect models' performance on long-context tasks?

• URL: http://arxiv.org/abs/2505.20276v3
• Date: Sat, 20 Sep 2025 06:44:40 GMT
• Title: Does quantization affect models' performance on long-context tasks?
• Abstract summary: Large language models (LLMs) now support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency.<n>We present the first systematic evaluation of quantized LLMs on tasks with long inputs (>64K tokens) and long-form outputs.<n>We find that, on average, 8-bit quantization preserves accuracy (0.8% drop), whereas 4-bit methods lead to substantial losses.
• Score: 31.289794345238544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) now support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. Quantization can mitigate these costs, but may degrade performance. In this work, we present the first systematic evaluation of quantized LLMs on tasks with long inputs (>64K tokens) and long-form outputs. Our evaluation spans 9.7K test examples, five quantization methods (FP8, GPTQ-int8, AWQ-int4, GPTQ-int4, BNB-nf4), and five models (Llama-3.1 8B and 70B; Qwen-2.5 7B, 32B, and 72B). We find that, on average, 8-bit quantization preserves accuracy (~0.8% drop), whereas 4-bit methods lead to substantial losses, especially for tasks involving long-context inputs (drops of up to 59%). This degradation tends to worsen when the input is in a language other than English. Crucially, the effects of quantization depend heavily on the quantization method, model, and task. For instance, while Qwen-2.5 72B remains robust under BNB-nf4, Llama-3.1 70B experiences a 32% performance drop on the same task. These findings highlight the importance of a careful, task-specific evaluation before deploying quantized LLMs, particularly in long-context scenarios and for languages other than English.

Related papers

• Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation [0.0]
  Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference.<n>It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla.<n>We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats.
  arXiv  Detail & Related papers  (2026-08-25T14:36:09Z)
• Studying quantization trade-offs for efficient inference deployment in machine translation [6.265237910291255]
  We study the quantization trade-offs of two translation model families, EuroLLM citepmartins2025eurollm and Hy-MT2 citepzheng2026hy across five models for efficient deployment on a single A100 or H100 GPU.
  arXiv  Detail & Related papers  (2026-07-31T13:15:11Z)
• K-Quantization and its Impact on Output Performance [1.1458853556386797]
  This paper investigates the performance of eight large language models (LLMs) at various quantization levels.<n>Findings indicate that while lower bit precision generally reduces performance, the impact varies across models and tasks.
  arXiv  Detail & Related papers  (2026-05-19T10:31:47Z)
• The Uneven Impact of Post-Training Quantization in Machine Translation [6.398727997282354]
  Quantization is essential for deploying large language models (LLMs) on resource-constrained hardware, but its implications for multilingual tasks remain underexplored.<n>We conduct the first large-scale evaluation of post-training quantization (PTQ) on machine translation across 55 languages using five LLMs ranging from 1.7B to 70B parameters.<n>Our analysis reveals that while 4-bit quantization often preserves translation quality for high-resource languages, significant degradation occurs for low-resource and typologically diverse languages, particularly in 2-bit settings.
  arXiv  Detail & Related papers  (2025-08-28T15:22:31Z)
• Low-bit Model Quantization for Deep Neural Networks: A Survey [123.89598730307208]
  This article surveys the recent five-year progress towards low-bit quantization on deep neural networks (DNNs)<n>We discuss and compare the state-of-the-art quantization methods and classify them into 8 main categories and 24 sub-categories according to their core techniques.<n>We shed light on the potential research opportunities in the field of model quantization.
  arXiv  Detail & Related papers  (2025-05-08T13:26:19Z)
• An Empirical Study of Qwen3 Quantization [30.214896404069677]
  Low-bit quantization presents a promising solution, yet its impact on Qwen3's performance remains underexplored.<n>We rigorously assess 5 existing classic post-training quantization techniques applied to Qwen3, spanning bit-widths from 1 to 8 bits.<n>Our findings reveal that while Qwen3 maintains competitive performance at moderate bit-widths, it experiences notable degradation in linguistic tasks under ultra-low precision.
  arXiv  Detail & Related papers  (2025-05-04T18:43:44Z)
• Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression [55.323397702682506]
  Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining.<n>We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery.
  arXiv  Detail & Related papers  (2025-04-10T02:19:03Z)
• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models [48.98109982725689]
  We conduct the first systematic study on quantized reasoning models.<n>Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths.<n>We identify model size, model origin, and task difficulty as critical determinants of performance.
  arXiv  Detail & Related papers  (2025-04-07T08:22:45Z)
• MBQ: Modality-Balanced Quantization for Large Vision-Language Models [20.018652727875367]
  Post-Training Quantization (PTQ) is an effective technique to reduce the memory and computation overhead.<n>Existing PTQ methods mainly focus on large language models (LLMs), without considering the differences across other modalities.<n>We propose Modality-Balanced Quantization (MBQ) for large vision-language models.
  arXiv  Detail & Related papers  (2024-12-27T07:55:36Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks [0.0]
  This study assesses the performance of five quantized code LLMs in Lua code generation tasks. The results suggest that the models quantized at the 4-bit integer precision offer the best trade-off between performance and model size. While quantization indeed increases the accessibility of smaller LLMs with 7 billion parameters, these LLMs demonstrate overall low performance.
  arXiv  Detail & Related papers  (2024-10-18T15:50:59Z)
• LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid [36.33062038680275]
  Large language models (LLMs) have shown immense potential across various domains. Post-training quantization has emerged as a promising technique to reduce memory requirements and decoding latency. We propose LeanQuant, a novel quantization method that is accurate, versatile, and scalable.
  arXiv  Detail & Related papers  (2024-07-14T00:23:51Z)
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [50.525259103219256]
  quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss.<n>We propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm.<n> EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP)
  arXiv  Detail & Related papers  (2024-07-10T17:53:30Z)
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [52.31791050376249]
  Quantization can accelerate large language model (LLM) inference.<n>We introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache.<n>QServe improves the maximum achievable serving of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen-721.5B by 2.4x on A100, 3.5x on L40S.
  arXiv  Detail & Related papers  (2024-05-07T17:59:30Z)
• Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization [62.15918574997175]
  It is known that language models contain outlier channels whose values on average are orders of magnitude higher than other channels. We propose a strategy which regularizes a layer's inputs via quantization-aware training (QAT) and its outputs via activation kurtosis regularization. We show that regularizing both the inputs and outputs is crucial for preventing a model's "migrating" the difficulty in input quantization to the weights.
  arXiv  Detail & Related papers  (2024-04-04T17:25:30Z)
• FPTQ: Fine-grained Post-Training Quantization for Large Language Models [28.11564378745513]
  We propose a novel W4A8 post-training quantization method for the available open-sourced LLMs. We obtain the state-of-the-art W4A8 quantized performance on BLOOM, LLaMA, and LLaMA-2 on standard benchmarks.
  arXiv  Detail & Related papers  (2023-08-30T12:18:18Z)
• Analyzing Quantization in TVM [0.0]
  TVM has the ability to quantize weights and support low-bit computations. 8-bit quantization is usually expected to achieve around 50% of the full-precision inference time. In this project, we thoroughly investigate the reasons behind the underperformance and assess the compatibility and optimization opportunities of 8-bit quantization in TVM.
  arXiv  Detail & Related papers  (2023-08-19T07:39:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.