Feature Evolution and Migration during Vision Transformer Training

• URL: http://arxiv.org/abs/2608.20134v1
• Date: Thu, 20 Aug 2026 15:00:21 GMT
• Title: Feature Evolution and Migration during Vision Transformer Training
• Abstract summary: We present a novel view on feature evolution in Vision Transformers (ViTs) by visualizing the training process over two dimensions -- network depth (layer) and training time (epochs)<n>We employ Sparse Autoencoders (SAEs) to extract candidate sparse features from CLS-token representations and compare their activation profiles across epoch--layer pairs.<n>We demonstrate how this framework of feature evolution allows us to describe feature migration, the change in the layer where a feature is most detectable during training.
• Score: 1.7966001353008776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel view on feature evolution in Vision Transformers (ViTs) by visualizing the training process over two dimensions -- network depth (layer) and training time (epochs). We employ Sparse Autoencoders (SAEs) to extract candidate sparse features from CLS-token representations and compare their activation profiles across epoch--layer pairs. This allows us to study feature-level dynamics that are not directly visible from representation-level similarity measures. Furthermore, we demonstrate how this framework of feature evolution allows us to describe feature migration, the change in the layer where a feature is most detectable during training. Our experiments show that migration is concentrated early in training, occurs more often toward earlier layers than toward deeper layers, and declines as feature organization stabilizes. We further find that deeper layers stabilize earlier and more strongly than shallow layers. The results show that our approach can be employed as a tool for understanding how ViTs learn and evolve.

Related papers

• IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers [6.663781005865136]
  Vision transformers (ViTs) have become the de facto standard for image encoding across many perception tasks.<n>We study how orientation selectivity emerges in ViTs by introducing a suite of neuroscience-inspired metrics.
  arXiv  Detail & Related papers  (2026-08-05T17:49:45Z)
• Gym-V: A Unified Vision Environment System for Agentic Vision Research [51.62136909074774]
  Gym-V is a unified platform of 179 procedurally generated visual environments across 10 domains with controllable difficulty.<n>We find that observation scaffolding is more decisive for training success than the choice of RL algorithm.<n>Cross-domain transfer experiments show that training on diverse task generalizes broadly while narrow training can cause negative transfer.
  arXiv  Detail & Related papers  (2026-03-16T15:37:07Z)
• Evolution of Concepts in Language Model Pre-Training [53.994470178155105]
  We track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders.<n>We find that most features begin to form around a specific point, while more complex patterns emerge in later training stages.
  arXiv  Detail & Related papers  (2025-09-21T18:53:12Z)
• Which Direction to Choose? An Analysis on the Representation Power of Self-Supervised ViTs in Downstream Tasks [43.473390101413166]
  Self-Supervised Learning for Vision Transformers (ViTs) has recently demonstrated considerable potential as a pre-training strategy for a variety of computer vision tasks.<n>This study aims to bridge the gap by systematically evaluating the use of unmodified features across image classification and segmentation tasks.
  arXiv  Detail & Related papers  (2025-09-18T11:46:07Z)
• Tracking the Feature Dynamics in LLM Training: A Mechanistic Study [11.64581123817424]
  We introduce SAE-Track, a novel method for efficiently obtaining a continual series of SAEs.<n>This study covers (2) the semantic evolution of features, (3) the underlying processes of feature formation, and (4) the directional drift of feature.<n>Our work provides new insights into the dynamics of features in large language models (LLMs), enhancing our understanding of training mechanisms and feature evolution.
  arXiv  Detail & Related papers  (2024-12-23T14:58:37Z)
• Locality Alignment Improves Vision-Language Models [55.275235524659905]
  Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors.<n>Our goal is to resolve this with a vision backbone that effectively captures both local and global image semantics.<n>We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed.
  arXiv  Detail & Related papers  (2024-10-14T21:01:01Z)
• What do Vision Transformers Learn? A Visual Exploration [68.50771218442776]
  Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision. This paper addresses the obstacles to performing visualizations on ViTs and explores the underlying differences between ViTs and CNNs. We also conduct large-scale visualizations on a range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin.
  arXiv  Detail & Related papers  (2022-12-13T16:55:12Z)
• Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learning [0.0]
  We study pretraining a Vision Transformer using several state-of-the-art self-supervised methods and assess the quality of the learned representations. Our results show that all methods are effective in learning useful representations and avoiding representational collapse. The encoder pretrained with the temporal order verification task shows the best results across all experiments.
  arXiv  Detail & Related papers  (2022-09-22T10:18:59Z)
• SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection [108.57862846523858]
  We revisit the self-supervised multi-task learning framework, proposing several updates to the original method. We modernize the 3D convolutional backbone by introducing multi-head self-attention modules. In our attempt to further improve the model, we study additional self-supervised learning tasks, such as predicting segmentation maps.
  arXiv  Detail & Related papers  (2022-07-16T19:25:41Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• Do Vision Transformers See Like Convolutional Neural Networks? [45.69780772718875]
  Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. Are they acting like convolutional networks, or learning entirely different visual representations? We find striking differences between the two architectures, such as ViT having more uniform representations across all layers.
  arXiv  Detail & Related papers  (2021-08-19T17:27:03Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.