The Asymmetric Harms of LLM Compression

• URL: http://arxiv.org/abs/2608.19670v1
• Date: Thu, 20 Aug 2026 06:06:14 GMT
• Title: The Asymmetric Harms of LLM Compression
• Abstract summary: Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts.<n>We systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias.
• Score: 9.45803162325743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) compression reduces deployment costs, but standard aggregate metrics like perplexity and accuracy often mask underlying behavioral shifts. In this work, we systematically evaluate 3 LLMs across 11 compression methods to investigate the effects of compression on knowledge retention, model confidence, and social bias. We find that compression disproportionately reduces the relative retention of head knowledge compared to tail knowledge. Furthermore, compressed models often remain substantially confident in their incorrect answers on newly lost knowledge. Finally, we demonstrate that stable aggregate bias scores can conceal substantial, opposing shifts in stereotypical preferences across demographic subgroups. Together, these findings reveal asymmetric behavioral changes that aggregate performance measures fail to capture, highlighting the need for granular evaluation of compressed models before deployment.

Related papers

• Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models [62.522141464687195]
  We introduce Action-JND, which extends the principle of just noticeable difference (JND) to embodied perception.<n>A token change is considered tolerable only when the induced action deviation remains within a tolerated margin.<n>The resulting action-tolerance score serves as a plug-and-play criterion for VLA compression paradigms.
  arXiv  Detail & Related papers  (2026-08-21T15:59:37Z)
• Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction [21.411353047208777]
  We benchmark 12 large language models (LLMs) under various compression configurations across five NLP tasks.<n>Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual.
  arXiv  Detail & Related papers  (2026-06-01T08:00:17Z)
• Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models [69.84867664371826]
  We show that visual token compression substantially degrades the robustness of Large Vision-Language Models (LVLMs)<n>Small and imperceptible perturbations can significantly alter token importance ranking, leading the compression mechanism to mistakenly discard task-critical information.<n>We propose a Compression-Aware Attack to systematically study and exploit this vulnerability.
  arXiv  Detail & Related papers  (2026-01-17T13:02:41Z)
• Decomposed Trust: Exploring Privacy, Adversarial Robustness, Fairness, and Ethics of Low-Rank LLMs [2.1311014724439845]
  We study how low-rank factorization affects trustworthiness across privacy, adversarial robustness, fairness, and ethical alignment.<n>Low-rank compression preserves or improves training data privacy but weakens PII protection during conversation.<n> adversarial robustness is generally preserved and often enhanced, even under deep compression.<n>Ethical reasoning degrades in zero-shot settings but partially recovers with few-shot prompting.
  arXiv  Detail & Related papers  (2025-11-27T04:40:56Z)
• Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion [56.12939353271623]
  From a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state.<n>We find this synchronicity is essentially the Compression Hacking'' in LM representations.<n>We propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline.
  arXiv  Detail & Related papers  (2025-05-23T12:11:03Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [56.00251589760559]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting.<n>We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.<n>Experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models [0.0]
  Large language models (LLMs) offer powerful capabilities but incur substantial computational costs. This study evaluates the impact of popular compression methods on the LLaMA-2-7B model. We show that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks.
  arXiv  Detail & Related papers  (2024-09-17T14:34:11Z)
• Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression [33.45167213570976]
  We investigate the impact of model compression on four dimensions: (1) degeneration harm, i.e., bias and toxicity in generation; (2) representational harm, i.e., biases in discriminative tasks; (3) dialect bias; and(4) language modeling and downstream task performance. Our analysis reveals that compression can lead to unexpected consequences.
  arXiv  Detail & Related papers  (2024-07-06T05:56:22Z)
• The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models [11.156816338995503]
  Large language models (LLMs) provide faster inference, smaller memory footprints, and enables local deployment. Two standard compression techniques are pruning and quantization, with the former eliminating redundant connections in model layers and the latter representing model parameters with fewer bits. Existing research on LLM compression primarily focuses on performance in terms of general metrics like perplexity or downstream task accuracy. More fine-grained metrics, such as those measuring parametric knowledge, remain significantly underexplored.
  arXiv  Detail & Related papers  (2023-12-01T22:27:12Z)
• What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [68.82486784654817]
  We study two popular model compression techniques including knowledge distillation and pruning. We show that compressed models are significantly less robust than their PLM counterparts on adversarial test sets. We develop a regularization strategy for model compression based on sample uncertainty.
  arXiv  Detail & Related papers  (2021-10-16T00:20:04Z)
• Learning, compression, and leakage: Minimising classification error via meta-universal compression principles [87.054014983402]
  A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding. Here we consider a NML-based decision strategy for supervised classification problems, and show that it attains PAC learning when applied to a wide variety of models. We show that the misclassification rate of our method is upper bounded by the maximal leakage, a recently proposed metric to quantify the potential of data leakage in privacy-sensitive scenarios.
  arXiv  Detail & Related papers  (2020-10-14T20:03:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.