Language Models Can Control Their Own Attention

• URL: http://arxiv.org/abs/2609.02737v1
• Date: Wed, 02 Sep 2026 15:43:38 GMT
• Title: Language Models Can Control Their Own Attention
• Abstract summary: We introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought.<n>DA significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale.
• Score: 42.07765767561011
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.

Related papers

• Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention [9.899649326522761]
  Multi-Token Localized Attention (MTLA) is a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim.<n>Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline.
  arXiv  Detail & Related papers  (2026-07-07T08:10:57Z)
• Bottleneck Tokens for Unified Multimodal Retrieval [16.707536543758344]
  Adapting decoder-only multimodal large language models (MLLMs) for unified multimodal retrieval faces two structural gaps.<n>First, existing methods rely on implicit pooling, which overloads the hidden state of a standard vocabulary token as the sequence-level representation.<n>Second, contrastive fine-tuning specifies what the embedding should match but provides no token-level guidance on how information should be compressed into it.<n>We introduce Bottleneck Tokens (BToks), a small set of learnable tokens that serve as a fixed-capacity explicit pooling mechanism.
  arXiv  Detail & Related papers  (2026-04-13T07:12:12Z)
• What am I missing here?: Evaluating Large Language Models for Masked Sentence Prediction [2.8514881296685113]
  Next Token Prediction (NTP) limits a model's ability to plan ahead or maintain long-range coherence.<n>We evaluate three commercial LLMs on Masked Sentence Prediction (MSP)<n>Our key finding reveals that commercial LLMs, despite their superlative performance in other tasks, are poor at predicting masked sentences in low-structured domains.
  arXiv  Detail & Related papers  (2025-08-11T07:25:50Z)
• AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models [14.013793473739236]
  We propose Adaptive holistic attention KV (Aha KV) to address the bias of the accumulated attention score.<n>Aha KV successfully mitigates bias and retains crucial tokens across global context.
  arXiv  Detail & Related papers  (2025-06-04T09:25:53Z)
• Learning to Attribute with Attention [75.61481181755744]
  We propose treating attention weights of different attention heads as features.<n>This way, we can learn how to effectively leverage attention weights for attribution.<n>Our resulting method, Attribution with Attention (AT2), reliably performs on par with approaches that involve many ablations.
  arXiv  Detail & Related papers  (2025-04-18T15:36:28Z)
• NoLiMa: Long-Context Evaluation Beyond Literal Matching [100.00398424275501]
  NoLiMa is a benchmark extending the needle-in-a-haystack (NIAH) test.<n>It requires models to infer latent associations to locate the needle within the haystack.<n>We evaluate 13 popular large language models that claim to support contexts of at least 128K tokens.
  arXiv  Detail & Related papers  (2025-02-07T18:49:46Z)
• Language Models (Mostly) Know When to Stop Reading [24.246459354913146]
  Large language models (LLMs) process entire input contexts indiscriminately, which is inefficient when the information required to answer a query is localized within the context.<n>We present dynamic context cutoff, a novel method enabling LLMs to self-terminate processing upon acquiring sufficient task-relevant information.
  arXiv  Detail & Related papers  (2025-02-03T03:38:29Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Core Context Aware Transformers for Long Context Language Modeling [50.774702091154204]
  We propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling.<n>Our method automatically focuses and strengthens core context while diminishing redundancy during the learning process.<n>Our method is able to replace the self-attention module in existing Large Language Models with minimal fine-tuning cost.
  arXiv  Detail & Related papers  (2024-12-17T01:54:08Z)
• Anti-LM Decoding for Zero-shot In-context Machine Translation [59.26037416204157]
  This work introduces an Anti-Language Model objective with a decay factor designed to address the weaknesses of In-context Machine Translation. We conduct experiments across 3 model types and sizes, 3 language directions, and for both greedy decoding and beam search.
  arXiv  Detail & Related papers  (2023-11-14T17:09:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.