What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

• URL: http://arxiv.org/abs/2608.18399v1
• Date: Wed, 19 Aug 2026 00:10:38 GMT
• Title: What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers
• Abstract summary: Vision transformers copy a pretrained teacher's attention maps.<n>They recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift.<n>What the copy delivers has never been measured directly in the attention structure and tied to robustness.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structure and tied to robustness. We build that instrumentation for ViT-S students of a self-supervised teacher on ImageNet-100, and report three findings that triangulate one conclusion. First, the transfer is essentially perfect and permanently so: the distilled student's attention ends up roughly two orders of magnitude closer to the teacher's than fine-tuning does, and does not drift with additional training. Second, the gap is real at 14$\times$ fewer parameters and 10$\times$ less data than previously studied, but it has a time axis. It tracks training maturity, and completing the schedules that the stopping rule interrupted closes it below our pre-registered threshold in two of three seeds, with comparisons at equal accuracy giving the same result. The endpoint gap at this scale is substantially a training-maturity artifact: robustness matures later than accuracy, and stopping rules tuned to accuracy undersample it. Third, forcing cross-row redundancy down by half the structural separation between the distilled and fine-tuned conditions produces no detectable robustness response under two registered ways of matching accuracy. Verified transfer, a gap that closes while the structure never moves, and a null under direct intervention are together consistent with the deficit residing in features, not in the visible attention structure. This is elimination plus intervention, and its scope is the regime we measured. In this regime, attention overlays show where a model looks, not what it knows.

Related papers

• Learning to Track from Privileged Target Appearances [67.80976117600649]
  We quantify a bottleneck with a non-deployable oracle that supplies an exact current-frame target crop.<n>We introduce Privileged Appearance Transfer for Tracking (PATT), a teacher-student training framework that transfers these privileged appearances to a deployable tracker.<n>PATT achieves consistent gains under both long- and short-term tracking protocols.
  arXiv  Detail & Related papers  (2026-09-02T11:43:43Z)
• Geometric Self-Distillation for Reasoning Generalization [12.38444431260744]
  On-policy distillation is a practical post-training recipe for large language models.<n>In privileged-context self-distillation, teacher and student are the same model conditioned on the same prefix.<n>We introduce GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior.
  arXiv  Detail & Related papers  (2026-07-07T23:16:19Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Perfect Demo Makes Poor Teacher: Learning Robust Alignment from Critical Motion Segments [53.85970147743299]
  Expert demonstrations are widely assumed to be the gold standard for robot imitation learning.<n>Yet for fine-grained manipulation such as insertion, stacking, and alignment, we uncover a counterintuitive failure mode: fluent demonstrations can be poor teachers.<n>We introduce STAIR, a compact dynamic feature that bridges the vision-language model and the action expert, distilling the short-horizon motion already recorded in each trajectory into dense, motion-aware supervision.
  arXiv  Detail & Related papers  (2026-06-14T04:15:29Z)
• Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions [4.672326975246762]
  Knowledge Distillation and mixup have proven effective at inducing smoothness in class boundaries.<n>Their interaction remains poorly understood, particularly when mixup is applied only during student training.<n>We show that this mismatch causes the teacher's supervisory signal to be dominated by distributional confusion.
  arXiv  Detail & Related papers  (2026-06-10T14:59:26Z)
• Phantom transitions in language model fine-tuning [0.0]
  Fine-tuning a language model on contexts with a near-synonym competitor often fails silently.<n>We study this regime across five transformer architectures spanning two families and a fivefold parameter range.<n>We observe sharp catapult-like jumps in the order parameter that resemble a phase transition.
  arXiv  Detail & Related papers  (2026-05-25T10:44:42Z)
• RAIL: Region-Aware Instructive Learning for Semi-Supervised Tooth Segmentation in CBCT [8.60376207420468]
  Region-Aware Instructive Learning (RAIL) is a dual-group dual-student, semi-supervised framework for CBCT tooth segmentation.<n>Our code will be available at https://github.com/Tournesol-Saturday/RAIL.
  arXiv  Detail & Related papers  (2025-05-06T13:50:57Z)
• Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks [50.2033914945157]
  We present a representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. The framework consists of three stages: estimating pairwise intensity functions, learning a projection which minimises a notion of intensity reconstruction error. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses.
  arXiv  Detail & Related papers  (2023-06-09T15:38:25Z)
• Bidirectional Semi-supervised Dual-branch CNN for Robust 3D Reconstruction of Stereo Endoscopic Images via Adaptive Cross and Parallel Supervisions [15.879059896662723]
  We propose a novel fashion of bidirectional learning between two learners, each of which can play both roles of teacher and student concurrently. Specifically, we introduce two self-supervisions, i.e., Adaptive Cross Supervision (ACS) and Adaptive Parallel Supervision (APS) The learned knowledge flows across branches along two directions: a cross direction (disparity guides distribution in ACS) and a parallel direction (disparity guides disparity in APS)
  arXiv  Detail & Related papers  (2022-10-15T13:30:41Z)
• Self-Point-Flow: Self-Supervised Scene Flow Estimation from Point Clouds with Optimal Transport and Random Walk [59.87525177207915]
  We develop a self-supervised method to establish correspondences between two point clouds to approximate scene flow. Our method achieves state-of-the-art performance among self-supervised learning methods.
  arXiv  Detail & Related papers  (2021-05-18T03:12:42Z)
• SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning [114.58986229852489]
  In this paper, we explore the basic and generic supervision in the sequence from spatial, sequential and temporal perspectives. We derive a particular form named Contrastive Learning (SeCo) SeCo shows superior results under the linear protocol on action recognition, untrimmed activity recognition and object tracking.
  arXiv  Detail & Related papers  (2020-08-03T15:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.