Attention Transfer Is Not Universally Effective for Vision Transformers

• URL: http://arxiv.org/abs/2605.07191v1
• Date: Fri, 08 May 2026 03:39:38 GMT
• Title: Attention Transfer Is Not Universally Effective for Vision Transformers
• Abstract summary: We revisit this finding on a benchmark of 20 teachers from 11 well-known ViT families.<n>While 7 families transfer successfully, 4 consistently fail, falling up to 5.1% below the from-scratch no-transfer baseline.<n>We identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism.
• Score: 47.26921741602587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard student ViT, is sufficient to recover the full benefit of the teacher's pre-trained weights. We revisit this finding on a comprehensive benchmark of 20 teachers from 11 well-known ViT families and reveal that Attention Transfer is not universally effective. While 7 families transfer successfully, 4 consistently fail, falling up to 5.1\% below the from-scratch no-transfer baseline. Further results demonstrate that this failure is family-consistent across model sizes, and persists under extended training durations, different transfer datasets, and out-of-distribution evaluations. Controlled analyses then consistently localize the problem to the attention-routing channel, indicating that the key issue is not whether the student can match the teacher's attention patterns, but whether the matched patterns remain functional for the student. Crucially, we identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism. By adding only the teacher's native architectural components to the student in a randomly initialized state, we completely reverse the failure for all 4 families. Notably, these components alone do not improve from-scratch training, confirming that they specifically unlock the usability of the teacher's attention. We further systematically show that this failure is not explained by the inadequate choice of transfer loss or by differences in pre-training recipes. Our findings refine the prevailing understanding of attention in ViT representations: attention is sufficient \textit{only} when the student architecture matches the teacher.

Related papers

• Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall [79.59596652272923]
  We propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident.<n>Relative to standard NTP, it achieves 1.61-1.71x the reasoning performance and 1.13-1.19x the knowledge and commonsense performance.
  arXiv  Detail & Related papers  (2026-09-01T17:00:30Z)
• What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers [0.0]
  Vision transformers copy a pretrained teacher's attention maps.<n>They recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift.<n>What the copy delivers has never been measured directly in the attention structure and tied to robustness.
  arXiv  Detail & Related papers  (2026-08-19T00:10:38Z)
• Framing the 5% Problem: Teachers' Perspectives on Persistence in Educational Technology [0.0]
  This study reports on a 90-minute participatory design workshop with 12 U.S. middle school mathematics teachers who use i-Ready Math weekly.<n>Thematic analysis identified four recurring dimensions of low persistence spanning motivation and buy-in, cognitive roadblocks, resilience under challenge, and contextual barriers.
  arXiv  Detail & Related papers  (2026-06-21T01:35:56Z)
• Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation [81.10000755917712]
  On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher.<n>As student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative.<n>To mitigate SFD, we introduce textbfLookahead Group Reward (ours).
  arXiv  Detail & Related papers  (2026-05-29T04:39:20Z)
• Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation [49.117085054884676]
  On-policy distillation trains a student model on its own rollouts using dense feedback from a stronger teacher.<n>We operationalize this principle through a trajectory-specific release rule.<n> Experimental results across strong-to-weak distillation tasks indicate that this release rule consistently outperforms standard full-trajectory OPD.
  arXiv  Detail & Related papers  (2026-05-13T15:05:30Z)
• From Correction to Mastery: Reinforced Distillation of Large Language Model Agents [13.982204994247718]
  Large Language Model agents excel at solving complex tasks through iterative reasoning and tool use.<n>Existing distillation approaches train smaller students to imitate full teacher trajectories.<n>We propose SCoRe, a student-centered framework in which the student generates training trajectories and the teacher corrects only the earliest error.
  arXiv  Detail & Related papers  (2025-09-12T15:34:07Z)
• Weak-to-Strong Generalization Even in Random Feature Networks, Provably [52.6431973649838]
  We show that weak-to-strong generalization does not require a strong learner like GPT-4.<n>We demonstrate, prove, and understand how the student can outperform the teacher, even though trained only on data labeled by the weak teacher.
  arXiv  Detail & Related papers  (2025-03-04T18:58:00Z)
• On the Surprising Effectiveness of Attention Transfer for Vision Transformers [118.83572030360843]
  Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. We investigate this question and find that the features and representations learned during pre-training are not essential.
  arXiv  Detail & Related papers  (2024-11-14T18:59:40Z)
• Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery [37.81825377491515]
  Generalized Category Discovery (GCD) involves a teacher-student framework in which the teacher imparts knowledge to the student to classify categories. GCD presents unique challenges, particularly the absence of priors for new classes, which can lead to the teacher's misguidance and unsynchronized learning with the student. We introduce FlipClass, a method that dynamically updates the teacher to align with the student's attention, instead of maintaining a static teacher reference.
  arXiv  Detail & Related papers  (2024-09-29T11:12:03Z)
• Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism [8.322293031346161]
  paradoxical studies indicate that closely replicating the teacher's behavior does not consistently improve student generalization. We propose this low-fidelity phenomenon as an underlying characteristic rather than a pathology when training KD.
  arXiv  Detail & Related papers  (2024-04-30T01:12:32Z)
• Random Teachers are Good Teachers [19.74244993871716]
  We investigate the implicit regularization induced by teacher-student learning dynamics in self-distillation. When distilling a student into such a random teacher, we observe a strong improvement of the distilled student over its teacher in terms of probing accuracy.
  arXiv  Detail & Related papers  (2023-02-23T15:26:08Z)
• Does Knowledge Distillation Really Work? [106.38447017262183]
  We show that while knowledge distillation can improve student generalization, it does not typically work as it is commonly understood. We identify difficulties in optimization as a key reason for why the student is unable to match the teacher.
  arXiv  Detail & Related papers  (2021-06-10T17:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.