Vision Transformers that Never Stop Learning

• URL: http://arxiv.org/abs/2603.07787v1
• Date: Sun, 08 Mar 2026 20:07:43 GMT
• Title: Vision Transformers that Never Stop Learning
• Abstract summary: We present a systematic investigation of loss of plasticity in Vision Transformers (ViTs)<n>Our analysis reveals that stacked attention modules exhibit increasing instability that exacerbates plasticity loss, while feed-forward network modules suffer even more pronounced degradation.<n>We propose ARROW, a geometry-aware model that preserves plasticity by adaptively reshaping directions using an online curvature estimate for the attention module.
• Score: 13.804234595369058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Loss of plasticity refers to the progressive inability of a model to adapt to new tasks and poses a fundamental challenge for continual learning. While this phenomenon has been extensively studied in homogeneous neural architectures, such as multilayer perceptrons, its mechanisms in structurally heterogeneous, attention-based models such as Vision Transformers (ViTs) remain underexplored. In this work, we present a systematic investigation of loss of plasticity in ViTs, including a fine-grained diagnosis using local metrics that capture parameter diversity and utilization. Our analysis reveals that stacked attention modules exhibit increasing instability that exacerbates plasticity loss, while feed-forward network modules suffer even more pronounced degradation. Furthermore, we evaluate several approaches for mitigating plasticity loss. The results indicate that methods based on parameter re-initialization fail to recover plasticity in ViTs, whereas approaches that explicitly regulate the update process are more effective. Motivated by this insight, we propose ARROW, a geometry-aware optimizer that preserves plasticity by adaptively reshaping gradient directions using an online curvature estimate for the attention module. Extensive experiments show that ARROW effectively improves plasticity and maintains better performance on newly encountered tasks.

Related papers

• The Rank and Gradient Lost in Non-stationarity: Sample Weight Decay for Mitigating Plasticity Loss in Reinforcement Learning [62.72522261010872]
  Deep reinforcement learning (RL) suffers from plasticity loss severely due to the nature of non-stationarity.<n>We study the plasticity loss problem from the theoretical perspective of network optimization.<n>We propose Sample Weight Decay -- a lightweight method to restore gradient magnitude.
  arXiv  Detail & Related papers  (2026-04-02T11:29:46Z)
• Rethinking Plasticity in Deep Reinforcement Learning [3.18807491942654]
  This paper investigates the fundamental mechanisms driving plasticity loss in deep reinforcement learning (RL)<n>We propose the Optimization-Centric Plasticity (OCP) hypothesis, which posits that plasticity loss arises because optimal points from previous tasks become poor local optima for new tasks.<n>Our findings provide a rigorous optimization-based framework for understanding and restoring network plasticity in complex domains.
  arXiv  Detail & Related papers  (2026-03-22T11:27:16Z)
• Plasticine: Accelerating Research in Plasticity-Motivated Deep Reinforcement Learning [122.67854581396578]
  Plasticine is an open-source framework for benchmarking plasticity optimization in deep reinforcement learning.<n>Plasticine provides single-file implementations of over 13 mitigation methods, 10 evaluation metrics, and learning scenarios.
  arXiv  Detail & Related papers  (2025-04-24T12:32:13Z)
• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment.<n>We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages [56.98243487769916]
  Plasticity, the ability of a neural network to evolve with new data, is crucial for high-performance and sample-efficient visual reinforcement learning. We propose Adaptive RR, which dynamically adjusts the replay ratio based on the critic's plasticity level.
  arXiv  Detail & Related papers  (2023-10-11T12:05:34Z)
• PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning [54.409634256153154]
  In Reinforcement Learning (RL), enhancing sample efficiency is crucial. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects.
  arXiv  Detail & Related papers  (2023-06-19T06:14:51Z)
• Understanding plasticity in neural networks [41.79540750236036]
  Plasticity is the ability of a neural network to quickly change its predictions in response to new information. Deep neural networks are known to lose plasticity over the course of training even in relatively simple learning problems.
  arXiv  Detail & Related papers  (2023-03-02T18:47:51Z)
• Unveiling the role of plasticity rules in reservoir computing [0.0]
  Reservoir Computing (RC) is an appealing approach in Machine Learning. We analyze the role that plasticity rules play on the changes that lead to a better performance of RC.
  arXiv  Detail & Related papers  (2021-01-14T19:55:30Z)
• Progressive Self-Guided Loss for Salient Object Detection [102.35488902433896]
  We present a progressive self-guided loss function to facilitate deep learning-based salient object detection in images. Our framework takes advantage of adaptively aggregated multi-scale features to locate and detect salient objects effectively.
  arXiv  Detail & Related papers  (2021-01-07T07:33:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.