When Less is More: The LLM Scaling Paradox in Context Compression

• URL: http://arxiv.org/abs/2602.09789v1
• Date: Tue, 10 Feb 2026 13:49:08 GMT
• Title: When Less is More: The LLM Scaling Paradox in Context Compression
• Abstract summary: We observe a Size-Fidelity Paradox: increasing the compressor size can lessen the faithfulness of reconstructed contexts.<n>We show that the culprit is not parameter count itself, but the excessive semantic capacity and amplified generative uncertainty that accompany scaling.
• Score: 14.27795208607174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling up model parameters has long been a prevalent training paradigm driven by the assumption that larger models yield superior generation capabilities. However, under lossy context compression in a compressor-decoder setup, we observe a Size-Fidelity Paradox: increasing the compressor size can lessen the faithfulness of reconstructed contexts though training loss decreases. Through extensive experiments across models from 0.6B to 90B, we coin this paradox arising from two dominant factors: 1) knowledge overwriting: larger models increasingly replace source facts with their own prior beliefs, e.g., ``the white strawberry'' $\to$ ``the red strawberry''; and 2) semantic drift: larger models tend to paraphrase or restructure content instead of reproducing it verbatim, e.g., ``Alice hit Bob'' $\to$ ``Bob hit Alice''. By holding model size fixed, we reflect on the emergent properties of compressed context representations. We show that the culprit is not parameter count itself, but the excessive semantic capacity and amplified generative uncertainty that accompany scaling. Specifically, the increased rank of context embeddings facilitates prior knowledge intrusion, whereas higher entropy over token prediction distributions promotes rewriting. Our results complement existing evaluations over context compression paradigm, underpinning a breakdown in scaling laws for faithful preservation in open-ended generation.

Related papers

• Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data [55.04392122319273]
  We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution.<n>The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart.
  arXiv  Detail & Related papers  (2026-07-13T17:58:50Z)
• PACE: Post-Causal Entropy Modeling for Learned LiDAR Point Cloud Compression [43.524231862354405]
  LiDAR point cloud compression is vital for autonomous systems to handle massive data from high-resolution sensors.<n>We propose PACE, a new framework that reformulates ancestral context aggregation as a non-causal backbone and causality to a lightweight, stage-scalable predictor.<n>Experiments demonstrate that PACE sets a new state-of-the-art in compression efficiency, achieving notable BD-BR savings and reducing decoding latency by over 90% in autoregressive mode.
  arXiv  Detail & Related papers  (2026-05-02T08:36:11Z)
• Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size [44.634649562117744]
  We formalize this apparent paradox through the first scaling laws for contextual entrainment.<n>We find entrainment follows predictable power-law scaling, but with opposite trends depending on context type.<n>Concretely, the largest models are four times more resistant to counterfactual misinformation than the smallest.
  arXiv  Detail & Related papers  (2026-04-14T20:12:05Z)
• Scale Dependent Data Duplication [29.59812821602787]
  We show that semantic duplicates operate increasingly like exact duplicates during training.<n>We embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m.<n>We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus.
  arXiv  Detail & Related papers  (2026-02-18T05:22:58Z)
• Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models [55.908141398092646]
  Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs.<n>We study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in the smaller models.<n>We show that small models such as $textttGPT2$ and $textttQwen3-0.6B$ exhibit severe condensation, whereas the larger models such as $textttGPT2-xl$ and $textttQwen3-32B
  arXiv  Detail & Related papers  (2026-01-30T16:07:03Z)
• On the Entropy Calibration of Language Models [52.47557449370603]
  We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text.<n>We find that the observed scaling behavior is similar to what is predicted by the simplified setting.<n>We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.
  arXiv  Detail & Related papers  (2025-11-15T00:33:03Z)
• Unified Scaling Laws for Compressed Representations [69.72517034565467]
  We investigate whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations.<n>Our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric.<n>We extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.
  arXiv  Detail & Related papers  (2025-06-02T16:52:51Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Diffusion Counterfactuals for Image Regressors [1.534667887016089]
  We present two methods to create counterfactual explanations for image regression tasks using diffusion-based generative models.<n>Both produce realistic, semantic, and smooth counterfactuals on CelebA-HQ and a synthetic data set.<n>We find that for regression counterfactuals, changes in features depend on the region of the predicted value.
  arXiv  Detail & Related papers  (2025-03-26T14:42:46Z)
• Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation [10.376875638696504]
  This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off.<n>We use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty.<n>We show that ACIP seamlessly complements common quantization-based compression techniques.
  arXiv  Detail & Related papers  (2025-02-03T18:40:58Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• On the Role of Optimization in Double Descent: A Least Squares Study [30.44215064390409]
  We show an excess risk bound for the descent gradient solution of the least squares objective. We find that in case of noiseless regression, double descent is explained solely by optimization-related quantities. We empirically explore if our predictions hold for neural networks.
  arXiv  Detail & Related papers  (2021-07-27T09:13:11Z)
• IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers [81.31885548824926]
  Self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. We present an Interpretability-Aware REDundancy REDuction framework (IA-RED$2$) We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4X speed-up.
  arXiv  Detail & Related papers  (2021-06-23T18:29:23Z)
• Reliable Model Compression via Label-Preservation-Aware Loss Functions [14.368823297066276]
  We present a framework that uses a teacher-student learning paradigm to better preserve labels. We obtain a significant reduction of up to 4.1X in the number of mismatches between the compressed and reference models.
  arXiv  Detail & Related papers  (2020-12-03T00:00:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.