2026-09-02 Daily Report: Language Models Can Control Their Own Attention

Language Models Can Control Their Own Attention

Authors Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

Affiliations Google / Korea Advanced Institute of Science and Technology

Categories Method / Attention / Controlling attention in decoding, Evaluation / Model Efficiency / Reducing token attendance with minimal accuracy drop, Application / Language Model Optimization / Improving decoding efficiency via attention control

License CC BY 4.0

Abstract Overview

This paper introduces Declarative Attention (DA), a prompting and inference protocol in which a language model explicitly declares where it needs to attend during decoding. DA partitions generation into three modes—global, focus, and local—and an inference engine parses these declarations to construct attention masks that skip most KV-cache reads without retraining the model. The method is evaluated zero-shot on off-the-shelf long-context models across 15 tasks, using a custom vLLM integration that applies block-aligned masking in global attention layers. The paper frames DA as an intrinsic alternative to external token-scoring approaches, aiming to reduce long-context decoding cost while keeping accuracy close to vanilla inference.

Novelty

The distinctive idea is to let the model state its own attention plan in text and use those control tokens to drive decode-time masking, rather than estimating sparse attention from proxy scores or hidden activations. The work also shows that this can be elicited zero-shot from standard models with a task-agnostic protocol and implemented in vLLM without changing attention kernels.

Results

Across 15 long-context tasks, DA reduces total attended tokens during decoding by 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B, with average accuracy drops of 1.27 and 2.75 percentage points, respectively. The paper attributes the efficiency gain to the attention mask itself, reports that stronger models narrow the accuracy gap, and shows that absolute token savings increase with context length. A roofline analysis further estimates decode wall-clock reductions to 0.71× of vanilla on Gemma-4-31B and 0.77× on Qwen-3.6-27B in optimized serving settings.

Key Points

DA organizes generation into explicit <global>, <focus>, and <local> modes so the model can control which context regions remain visible during decoding.
The chunked DA prompt format alone is nearly accuracy-preserving, while the custom attention mask is the main source of the observed token savings.
Accuracy under DA improves with model scale, and the largest absolute savings appear on the longest-context tasks such as code repositories and dialogue history.

References

arXiv: https://arxiv.org/abs/2609.02737v1
Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.02737v1
Hugging Face Papers: https://huggingface.co/papers/2609.02737