Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off

• URL: http://arxiv.org/abs/2608.15459v1
• Date: Sun, 16 Aug 2026 00:37:38 GMT
• Title: Not All Attention Is Equal: A Quantitative Survey of the EEI Trade-off
• Abstract summary: This survey covers four connected threads: their formulation for sequence-to-sequence tasks, adaptation to computer vision, efficiency innovations that address the quadratic bottleneck, and advances in interpretability.<n>We define three criteria: efficiency, expressiveness, and interpretability, and compare twenty-one methods using an EEI scoring framework.<n>We conclude by framing attention research as an expansion of the efficiency-expressiveness-interpretability frontier and identifying future directions.
• Score: 8.068122032539332
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Attention mechanisms have driven machine learning for a decade, from neural machine translation to language models that do general-purpose reasoning. This survey covers four connected threads: their formulation for sequence-to-sequence tasks, adaptation to computer vision, efficiency innovations that address the quadratic bottleneck, and advances in interpretability. We define three criteria: efficiency, expressiveness, and interpretability, and compare twenty-one methods using an EEI scoring framework. Scores come from a single rater with an assumed +/-1-point perturbation range. A deterministic Monte Carlo analysis with 200,000 samples shows that, under this perturbation model, rank changes of more than one position occur in 67-70% of samples on average. A rank-matched null model reproduces a similar stability profile, so the results support coarse tier-level comparisons rather than fine-grained rankings. The survey traces attention from Bahdanau-Luong alignment through the Transformer and into vision architectures. It reviews fixed and learned sparse attention, linear attention, IO-aware exact algorithms including FlashAttention, and state-space alternatives including Mamba. It also covers induction heads, superposition, and the attention-SSM duality. We further provide a structured narrative review, a benchmark synthesis with cross-study caveats, a five-problem research gap analysis, and a 2015-2026 evolution timeline. We conclude by framing attention research as an expansion of the efficiency-expressiveness-interpretability frontier and identifying future directions including unified efficiency benchmarks, learned routing for hybrid architectures, length generalization, and scalable mechanistic interpretability.

Related papers

• Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B [51.10897657510313]
  Large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction.<n>We investigate Llama 3.1-8B from a mechanistic interpretability point of view.<n>We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure.
  arXiv  Detail & Related papers  (2026-08-19T01:09:51Z)
• Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation [0.0]
  AQDE holds growing promise for educational assessment because it has the potential to yield difficulty estimates competitive with expert judgment.<n>Prior AQDE studies report mixed evidence on whether adding distractors as additional text to the question stem and the correct key consistently improves difficulty prediction.<n>We hypothesize that the effectiveness of distractor information depends on its structural representation, and that explicitly modeling distractors as separate components improves difficulty estimation over baselines that omit this information.
  arXiv  Detail & Related papers  (2026-06-08T03:35:49Z)
• Capability $\neq$ Interpretability: Human Interpretability of Vision Foundation Models [24.250649573452666]
  We build a framework for measuring and comparing the human interpretability of vision models.<n>A chance-anchored scoring function places every model on a common scale.<n>Foundation models are consistently *less* interpretable than their supervised counterparts.
  arXiv  Detail & Related papers  (2026-05-19T18:00:06Z)
• Tree-of-Evidence: Efficient "System 2" Search for Faithful Multimodal Grounding [1.6365003324232734]
  We introduce Tree-of-Evidence (ToE), an inference-time search algorithm that frames interpretability as a discrete optimization problem.<n>ToE employs lightweight Evidence Bottlenecks that score coarse groups or units of data and performs a beam search to identify the compact evidence set required to reproduce the model's prediction.<n>Under sparse evidence budgets, ToE achieves higher decision agreement and lower probability fidelity error than other approaches.
  arXiv  Detail & Related papers  (2026-04-09T01:28:14Z)
• Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking [40.652380579951206]
  In-Context Re-ranking (ICR) has recently been proposed as an $O(1)$ alternative method.<n>ICR extracts internal attention signals directly, avoiding the overhead of text generation.<n>No unified study has compared internal attention with traditional generative and likelihood-based mechanisms.
  arXiv  Detail & Related papers  (2026-02-26T03:51:31Z)
• ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMs [37.23311145049677]
  We present ReLE, a scalable system designed to diagnose Capability Anisotropy.<n>We evaluate 304 models across a Domain $times$ Capability Symbolic matrix comprising 207,843 samples.
  arXiv  Detail & Related papers  (2026-01-24T09:57:59Z)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
  We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
  arXiv  Detail & Related papers  (2025-09-29T15:07:28Z)
• The Lie of the Average: How Class Incremental Learning Evaluation Deceives You? [48.83567710215299]
  Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones.<n>We argue that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution.<n>We propose EDGE, an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity.
  arXiv  Detail & Related papers  (2025-09-26T17:00:15Z)
• Long-Sequence Recommendation Models Need Decoupled Embeddings [49.410906935283585]
  We identify and characterize a neglected deficiency in existing long-sequence recommendation models.<n>A single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes.<n>We propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are learned separately to fully decouple attention and representation.
  arXiv  Detail & Related papers  (2024-10-03T15:45:15Z)
• Balanced Classification: A Unified Framework for Long-Tailed Object Detection [74.94216414011326]
  Conventional detectors suffer from performance degradation when dealing with long-tailed data due to a classification bias towards the majority head categories. We introduce a unified framework called BAlanced CLassification (BACL), which enables adaptive rectification of inequalities caused by disparities in category distribution. BACL consistently achieves performance improvements across various datasets with different backbones and architectures.
  arXiv  Detail & Related papers  (2023-08-04T09:11:07Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.