The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

• URL: http://arxiv.org/abs/2606.11198v1
• Date: Tue, 21 Apr 2026 08:47:00 GMT
• Title: The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
• Abstract summary: We formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples.<n>KG triples capture 2-3x more attention per token than semantically equivalent natural-language text.<n>We develop a formal framework decomposing attention scores into semantic and structural components.
• Score: 11.226669792850645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution. We identify and formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples, due to their relational delimiters and repeated slot patterns, capture 2-3x more attention per token than semantically equivalent natural-language text ($\hat{o}$(KG) $\approx$ 0.70 vs. $\hat{o}$(neutral) $\approx$ 0.25), compressing demonstration attention by up to 42% -- regardless of whether the triples are relevant or noise. We develop a formal framework decomposing attention scores into semantic and structural components (Eq. 2), derive a compression bound (Proposition 1) connecting token-level format bias to demonstration attention loss, and show that the structural term governs how much attention is diverted while the semantic term governs whether this helps or hurts. This decoupling reveals two orthogonal axes for improving retrieval-augmented ICL: optimising retrieval quality (semantic axis) and reducing format-driven attention capture (structural axis). Empirically, across two model families (Mistral-7B, LLaMA-3-8B) and three QA benchmarks, we observe that source-task alignment dominates: task-matched BM25 retrieval achieves 58-62% on HotpotQA vs. ConceptNet's 25-27%, a >30 pp gap that dwarfs all gating strategies ($\leq$2 pp). We derive five structure-aware mitigation strategies from the framework, ranging from zero-cost prompt modifications to training-time regularisation; format flattening (S3) is validated by both accuracy and attention-level evidence from a verbalized-triple control, while structural dispersal (S1) yields mixed results that illuminate the challenges of format-level intervention.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• Mapping and Measuring the Behavioral Evolution of Large Language Models [28.603443859513153]
  Benchmark leaderboards summarize how well a language model performs, but not how its behavior relates to that of other models or changes across generations.<n>We characterize the output behavior of 32 models from six families using their responses to a shared bank of 10,000 prompts.
  arXiv  Detail & Related papers  (2026-08-11T15:07:28Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models [16.39553879497114]
  SPpruner is a subject-centric progressive reduction paradigm.<n>It emulates the textitFocus-then-Context mechanism of the human visual perception system.<n>It can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input.
  arXiv  Detail & Related papers  (2026-05-20T09:37:53Z)
• SLASH the Sink: Sharpening Structural Attention Inside LLMs [55.75072688988806]
  Large Language Models (LLMs) show remarkable semantic understanding but struggle with structural understanding when processing graph topologies in a serialized format.<n>We propose StructuraL Attention SHarpening (SLASH), which amplifies this internal structural understanding via a plug-and-play attention redistribution.
  arXiv  Detail & Related papers  (2026-05-11T12:59:07Z)
• STaR-DRO: Stateful Tsallis Reweighting for Group-Robust Structured Prediction [0.20165644958637055]
  We present a two-part framework for controllable inference and robust fine-tuning.<n>First, we introduce a task-agnostic prompting strategy to address format drift, label ambiguity, evidence hallucination, and metadata-conditioned confusion.<n>Second, we introduce STaR-DRO, a stateful robust optimization method for group heterogeneity.
  arXiv  Detail & Related papers  (2026-04-09T21:58:42Z)
• SEER: Spectral Entropy Encoding of Roles for Context-Aware Attention-Based Design Pattern Detection [0.0]
  This paper presents an upgraded version of our prior method Context Is All You Need for detecting Gang of Four (GoF) design patterns from source code.<n> SEER addresses these limitations with two principled additions: (i) a spectral-entropy role encoder that derives per-member role embeddings from the Laplacian spectrum of each class's interaction graph, and (ii) a time-weighted calling context that assigns empirically calibrated duration priors to method categories.<n>We evaluate SEER on PyDesignNet (1,832 files, 35,000 sequences, 23 GoF patterns) and observe consistent gains over our previous system
  arXiv  Detail & Related papers  (2026-01-19T19:13:40Z)
• Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression [55.51959317490934]
  Large language models (LLMs) have demonstrated promising capabilities in Text-Attributed Graph (TAG) understanding.<n>We argue that graphs inherently contain rich structural and semantic information, and that their effective exploitation can unlock potential gains in LLMs reasoning performance.<n>We propose Homophily-aware Structural and Semantic Compression for LLMs (HS2C), a framework centered on exploiting graph homophily.
  arXiv  Detail & Related papers  (2026-01-13T03:35:18Z)
• Spectral Archaeology: The Causal Topology of Model Evolution [0.0]
  Behavioral benchmarks tell us textitwhat a model does, but not textithow.<n>We introduce a training-free mechanistic probe using attention-graph spectra.<n>Across 12 models and 10 languages, these measures yield stable fingerprints'' that expose discontinuities missed by standard evaluation.
  arXiv  Detail & Related papers  (2026-01-06T21:26:54Z)
• Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models [44.28116882776357]
  We present textbfPunctuation-aware textbfHybrid textbfSparse textbfAttention textbf(PHSA), a trainable sparse attention framework that leverages punctuation tokens as semantic boundary anchors.<n>Specifically, we design a dual-branch aggregation mechanism that fuses global semantic representations with punctuation-enhanced boundary features, preserving the core semantic structure while introducing almost no additional computational overhead.
  arXiv  Detail & Related papers  (2026-01-06T08:47:16Z)
• CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks [53.88194225946438]
  Chain-of-Thought for Detection (CoT4Det) is a simple but efficient strategy that reformulates perception tasks into three interpretable steps.<n>We show that CoT4Det significantly improves perception performance without compromising general vision language capabilities.
  arXiv  Detail & Related papers  (2025-12-07T05:26:30Z)
• TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework [62.66056331998838]
  TeaRAG is a token-efficient agentic RAG framework capable of compressing both retrieval content and reasoning steps.<n>Our reward function evaluates the knowledge sufficiency by a knowledge matching mechanism, while penalizing excessive reasoning steps.
  arXiv  Detail & Related papers  (2025-11-07T16:08:34Z)
• Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models [63.64507678113921]
  We introduce Graph-KV, which governs interaction through structural inductive biases.<n>In this framework, 'target' segments selectively attend only to the KV-cache of their designated'source' segments.<n>We evaluate Graph-KV across three scenarios: (1) seven RAG benchmarks spanning direct inference, multi-hop reasoning, and long-document understanding; (2) Arxiv-QA, a novel academic paper QA task with full-text scientific papers structured as citation ego-graphs; and (3) paper topic classification within a citation network.
  arXiv  Detail & Related papers  (2025-06-09T00:30:08Z)
• S2-Attention: Hardware-Aware Context Sharding Among Attention Heads [49.1454481007861]
  Sparse attention selectively attends to a subset of tokens in the context.<n>It remains unclear whether sparse attention can maintain the model's quality at a scale of today's large language models.<n>This paper presents Sparsely-Sharded(S2) Attention, a Triton library that provides kernel optimization for sparse attention customizable at both per-head and per-context-range levels.
  arXiv  Detail & Related papers  (2024-07-25T00:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.