Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation

• URL: http://arxiv.org/abs/2607.02593v1
• Date: Wed, 01 Jul 2026 08:03:30 GMT
• Title: Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
• Abstract summary: We introduce Token-level Response-visual Attention Guidance (TRAG)<n>TRAG shifts the focus to response-to-vision signals and employs token-specific objectives.<n>Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines.
• Score: 36.9736177938003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promising, the precise utility of which attention signals to distill remains under-explored. In this work, we challenge the conventional reliance on prompt-to-vision attention by revealing that downstream performance correlates strongly with response-to-vision attention similarity to the teacher, but negligibly with that of prompt-conditioned attention. Furthermore, we observe that attention distributions exhibit significant variance across individual tokens, indicating that a uniform distillation objective is suboptimal. To this end, we introduce Token-level Response-visual Attention Guidance (TRAG), a distillation objective that 1) shifts the focus to response-to-vision signals and 2) employs token-specific objectives by adaptively weighting the Kullback-Leibler divergence based on attention entropy, effectively guiding the student to mirror the teacher's precise visual focus. Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines.

Related papers

• LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression [50.454633334898396]
  Large vision-language models (LVLMs) exhibit strong reasoning ability but suffer from visual forgetting during long-horizon decoding.<n>We propose LASER, a post-training framework that regulates both the visual attention trajectory and intra-visual token attention distribution during reasoning.<n>Experiments on eight benchmark datasets demonstrate that LASER consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2026-07-02T04:59:56Z)
• Accelerating Multimodal Large Language Models with Prior-Corrected Token Reduction [42.23525789989544]
  We show that attention is often dominated by a model-induced prior in Multimodal Large Language Models (MLLMs)<n>We propose Prior-Corrected Token Reduction (PriorTR) to separate task-conditioned attention from the model-induced prior.<n>PriorTR consistently improves the trade-off between accuracy and efficiency over strong training-free baselines.
  arXiv  Detail & Related papers  (2026-06-23T05:24:50Z)
• From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning [72.4876727619987]
  We find that reasoning performance is strongly correlated with Visual Attention Score (VAS)<n>To validate its causal role, we design training-free interventions that directly modulate attention allocation during inference.<n>We propose Attention-Guided Visual Anchoring and Reflection, a comprehensive cold-start framework that integrates visual-anchored data synthesis, attention-guided objectives, and visual-anchored reward shaping.
  arXiv  Detail & Related papers  (2026-03-04T08:22:27Z)
• A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts [80.98474052840929]
  Gated attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard attention.<n>We show that each entry in a gated attention matrix or a multi-head self-attention matrix can be written as a hierarchical mixture of experts.
  arXiv  Detail & Related papers  (2026-02-01T22:22:13Z)
• Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruning [62.23671919314693]
  Large language models (LLMs) have demonstrated significant improvements in contextual understanding.<n>However, their ability to attend to truly critical information during long-context reasoning and generation still falls behind the pace.<n>We introduce a two-stage framework called Learning to Focus (LeaF) to mitigate confounding factors.
  arXiv  Detail & Related papers  (2025-06-09T15:16:39Z)
• Don't Take Things Out of Context: Attention Intervention for Enhancing Chain-of-Thought Reasoning in Large Language Models [32.71672086718058]
  Few-shot Chain-of-Thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs)<n>We observe that isolated segments, words, or tokens within CoT demonstrations can unexpectedly disrupt the generation process of LLMs.<n>We propose a Few-shot Attention Intervention method (FAI) that dynamically analyzes the attention patterns of demonstrations to accurately identify these tokens.
  arXiv  Detail & Related papers  (2025-03-14T07:46:33Z)
• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration [15.36841874118801]
  We aim to provide a more profound understanding of the existence of attention sinks within large language models (LLMs) We propose a training-free Attention Technique (ACT) that automatically optimize the attention distributions on the fly during inference in an input-adaptive manner. ACT achieves an average improvement of up to 7.30% in accuracy across different datasets when applied to Llama-30B.
  arXiv  Detail & Related papers  (2024-06-22T07:00:43Z)
• Using Early Readouts to Mediate Featural Bias in Distillation [30.5299408494168]
  Deep networks tend to learn spurious feature-label correlations in real-world supervised learning tasks. We propose a novel early readout mechanism whereby we attempt to predict the label using representations from earlier network layers.
  arXiv  Detail & Related papers  (2023-10-28T04:58:15Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.