Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

• URL: http://arxiv.org/abs/2510.06477v1
• Date: Tue, 07 Oct 2025 21:27:24 GMT
• Title: Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• Abstract summary: We present a surprising connection between attention sinks and compression valleys, tracing both to the formation of massive activations in the residual stream.<n>We propose the Mix-Compress-Refine theory of information flow, as an attempt to explain how LLMs organize their computation in depth by controlling attention and representational compression via massive activations.
• Score: 41.97400057649291
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, we present a surprising connection between attention sinks and compression valleys, tracing both to the formation of massive activations in the residual stream. We prove theoretically that massive activations necessarily produce representational compression and establish bounds on the resulting entropy reduction. Through experiments across several models (410M-120B parameters), we confirm that when the beginning-of-sequence token develops extreme activation norms in the middle layers, both compression valleys and attention sinks emerge simultaneously. Targeted ablation studies validate our theoretical predictions. This unified view motivates us to propose the Mix-Compress-Refine theory of information flow, as an attempt to explain how LLMs organize their computation in depth by controlling attention and representational compression via massive activations. Specifically, we posit that Transformer-based LLMs process tokens in three distinct phases: (1) broad mixing in the early layers, (2) compressed computation with limited mixing in the middle layers, and (3) selective refinement in the late layers. Our framework helps explain why embedding tasks perform best at intermediate layers, whereas generation tasks benefit from full-depth processing, clarifying differences in task-dependent representations.

Related papers

• TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization [15.401450705732232]
  Ternarization has emerged as a promising compression technique.<n>Existing methods struggle with heavy-tailed activation distributions.<n>We propose TWLA, a framework that achieves 1.58-bit weight compression and 4-bit activation quantization.
  arXiv  Detail & Related papers  (2026-06-11T08:37:20Z)
• A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models [56.03096341469333]
  We investigate the origins of massive activations in large language models (LLMs)<n>We identify a specific layer named the textbfMassive Emergence Layer (ME Layer), that is consistently observed across model families.<n>We show that, within the ME Layer both the RMSNorm and the FFN parameters jointly contribute to the emergence of massive activations.<n>Motivated by this limitation, we propose a simple and effective method to reduce the rigidity of the massive activation token.
  arXiv  Detail & Related papers  (2026-05-08T21:37:27Z)
• When Does Sparsity Mitigate the Curse of Depth in LLMs [53.137717161619484]
  We show that sparsity acts as a regulator of variance propagation and thereby improves depth utilization.<n>Our results reveal sparsity as a key yet previously overlooked mechanism for effective depth scaling in large language models.
  arXiv  Detail & Related papers  (2026-03-16T15:04:16Z)
• PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training [5.275001711555517]
  We propose Principal-Random Subspace for LLM Activation Compression (PRAC)<n>PRAC decomposes activations into two components: a principal subspace captured via SVD to retain dominant information, and a random subspace sampled from the orthogonal complement to approximate the tail.<n>Experiments on pre-training and fine-tuning tasks demonstrate that PRAC achieves up to 36% total memory reduction with negligible performance degradation and minimal computational cost.
  arXiv  Detail & Related papers  (2026-02-26T15:23:34Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Beyond Surface Reasoning: Unveiling the True Long Chain-of-Thought Capacity of Diffusion Large Language Models [54.81955614221652]
  parallel decoding, which enables simultaneous token updates, conflicts with the causal order often required for rigorous reasoning.<n> Behavioral analyses in both simple and complex reasoning tasks show thatDLLMs exhibit genuine parallelism only for directly decidable outputs.<n>We propose several practical mitigations, parallel-oriented prompting, diffusion early stopping, and parallel scaling, to reduce PSC-induced ineffectiveness and inefficiencies.
  arXiv  Detail & Related papers  (2025-10-10T16:58:14Z)
• Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility [90.894232610821]
  We analyze Transformers through the lens of rank structure.<n>We show that time-series embeddings exhibit sharply decaying singular value spectra.<n>We prove that the associated $Q/K/V$ projections admit accurate low-rank approximations.
  arXiv  Detail & Related papers  (2025-10-02T23:56:17Z)
• EARN: Efficient Inference Acceleration for LLM-based Generative Recommendation by Register Tokens [47.60523011706102]
  Large Language Model-based generative recommendation (LLMRec) has achieved notable success, but it suffers from high inference latency.<n>We propose EARN, an efficient inference framework that leverages the early layers to compress information into register tokens placed at the input sequence boundaries.
  arXiv  Detail & Related papers  (2025-07-01T12:42:06Z)
• A Two-Phase Perspective on Deep Learning Dynamics [0.0]
  We propose that learning in deep neural networks proceeds in two phases: a rapid curve fitting phase followed by a slower compression or coarse graining phase.<n>We empirically show that the associated timescales align in two rather different settings.<n>We argue that the second phase is not actively optimized by standard training algorithms and may be unnecessarily prolonged.
  arXiv  Detail & Related papers  (2025-04-17T06:57:37Z)
• Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs [77.66717051042032]
  Practitioners have consistently observed three puzzling phenomena in transformer-based large language models. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights. We elucidate the mechanisms behind extreme-token phenomena.
  arXiv  Detail & Related papers  (2024-10-17T17:54:06Z)
• Exploring Activation Patterns of Parameters in Language Models [27.454051736471374]
  We propose a gradient-based metric to assess the activation level of model parameters. We find that parameters in the shallow layers will be activated densely, which means a larger portion of parameters will have great impacts on the outputs. In deep layers, the similarity of the distributions of activated parameters is positively correlated to the empirical data relevance.
  arXiv  Detail & Related papers  (2024-05-28T03:49:54Z)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion [83.90492831583997]
  We show that a batch-normalized network can keep the optimal signal propagation properties, but avoid exploding gradients in depth. We use a Multi-Layer Perceptron (MLP) with linear activations and batch-normalization that provably has bounded depth. We also design an activation shaping scheme that empirically achieves the same properties for certain non-linear activations.
  arXiv  Detail & Related papers  (2023-10-03T12:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.