Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

• URL: http://arxiv.org/abs/2608.28383v1
• Date: Fri, 28 Aug 2026 14:40:47 GMT
• Title: Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs
• Abstract summary: We study ViT attention heads and find they differentiate into object-specialist roles, a pattern most pronounced under full attention.<n>We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance.<n>We design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute.
• Score: 53.38240440614346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.

Related papers

• ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs [32.51643420504522]
  Multimodal Large Language Models (MLLMs) are critically hampered by hallucination.<n>We identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention.<n>We propose ADAPT, an attention-based framework that intervenes directly on text-to-image cross-attention dynamics.
  arXiv  Detail & Related papers  (2026-06-30T02:46:10Z)
• HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization [54.02338209487246]
  We introduce HydraHead, a novel architecture that hybridizes Full Attention (FA) and Linear Attention (LA) along the head axis.<n> HydraHead features an interpretability-driven selection strategy that identifies retrieval-critical heads and preserves FA only for them.<n>It achieves over 69% improvement over the baseline at 512K context length, approaching Qwen3.5, a leading model of comparable size with a native context length of 256K.
  arXiv  Detail & Related papers  (2026-06-18T11:14:38Z)
• ViT-AdaLA: Adapting Vision Transformers with Linear Attention [71.36851471416034]
  Vision Transformers (ViTs) based vision foundation models (VFMs) have achieved remarkable performance across diverse vision tasks.<n>Existing linear attention approaches for ViTs are typically trained from scratch, requiring substantial computational resources.<n>We propose ViT-AdaLA, a novel framework for effectively adapting and transferring prior knowledge from VFMs to linear attention ViTs.
  arXiv  Detail & Related papers  (2026-03-17T02:15:48Z)
• MMSpec: Benchmarking Speculative Decoding for Vision-Language Models [43.4716250339058]
  We introduce MMSpec, the first benchmark for evaluating speculative decoding in vision-language models.<n> MMSpec contains 600 multimodal samples across six task categories and integrates ten representative speculative decoding algorithms.<n>We propose ViSkip, a plug-and-play speculative decoding method that dynamically adapts speculation to vision tokens and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-16T08:55:42Z)
• A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention [13.144793724034761]
  Transformers serve as the foundation of most modern large language models.<n>A fundamental gap remains: their expressive power relative to full attention lacks a rigorous theoretical characterization.<n>Our work provides the first provable separation between hybrid attention and standard full attention, offering a theoretical perspective for understanding the fundamental capabilities and limitations of different attention mechanisms.
  arXiv  Detail & Related papers  (2026-02-02T07:47:21Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• Scratching Visual Transformer's Back with Uniform Attention [49.31658397277271]
  We study the role of the density of the attention in ViT models.<n>We manually insert the uniform attention to each layer of ViT models to supply the much needed dense interactions.<n>We observe that the inclusion of CB reduces the degree of density in the original attention maps and increases both the capacity and generalizability of the ViT models.
  arXiv  Detail & Related papers  (2022-10-16T06:14:27Z)
• Refiner: Refining Self-attention for Vision Transformers [85.80887884154427]
  Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs. We introduce a conceptually simple scheme, called refiner, to directly refine the self-attention maps of ViTs. refiner applies convolutions to augment local patterns of the attention maps, which we show is equivalent to a distributed local attention features are aggregated locally with learnable kernels and then globally aggregated with self-attention.
  arXiv  Detail & Related papers  (2021-06-07T15:24:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.