Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification

• URL: http://arxiv.org/abs/2508.07577v1
• Date: Mon, 11 Aug 2025 03:18:47 GMT
• Title: Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification
• Abstract summary: We show that shifts in LayerNorm parameters after fine-tuning are indicative of the transitions between source and target domains.<n>Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.
• Score: 30.16664767564679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LayerNorm is pivotal in Vision Transformers (ViTs), yet its fine-tuning dynamics under data scarcity and domain shifts remain underexplored. This paper shows that shifts in LayerNorm parameters after fine-tuning (LayerNorm shifts) are indicative of the transitions between source and target domains; its efficacy is contingent upon the degree to which the target training samples accurately represent the target domain, as quantified by our proposed Fine-tuning Shift Ratio ($FSR$). Building on this, we propose a simple yet effective rescaling mechanism using a scalar $\lambda$ that is negatively correlated to $FSR$ to align learned LayerNorm shifts with those ideal shifts achieved under fully representative data, combined with a cyclic framework that further enhances the LayerNorm fine-tuning. Extensive experiments across natural and pathological images, in both in-distribution (ID) and out-of-distribution (OOD) settings, and various target training sample regimes validate our framework. Notably, OOD tasks tend to yield lower $FSR$ and higher $\lambda$ in comparison to ID cases, especially with scarce data, indicating under-represented target training samples. Moreover, ViTFs fine-tuned on pathological data behave more like ID settings, favoring conservative LayerNorm updates. Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.

Related papers

• Geometric Characterisation and Structured Trajectory Surrogates for Clinical Dataset Condensation [1.6261700844888172]
  Trajectory matching (TM) is a widely used condensation approach that supervises synthetic data using changes in model parameters observed during training on real data.<n>We show that a fixed synthetic dataset can only reproduce a limited span of such training-induced parameter changes.<n>Motivated by this mismatch, we propose Bezier Trajectory Matching (BTM), which replaces SGD trajectories with quadratic Bezier trajectory surrogates.
  arXiv  Detail & Related papers  (2026-04-23T12:53:46Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Open-World Test-Time Adaptation with Hierarchical Feature Aggregation and Attention Affine [17.151364853811128]
  Test-time adaptation (TTA) refers to adjusting the model during the testing phase to cope with changes in sample distribution.<n>We propose a Hierarchical Ladder Network that extracts OOD features from class tokens aggregated across all Transformer layers.<n>We also introduce an Attention Affine Network (AAN) that adaptively refines the self-attention mechanism conditioned on the token information to better adapt to domain drift.
  arXiv  Detail & Related papers  (2025-11-16T14:05:23Z)
• HyperTTA: Test-Time Adaptation for Hyperspectral Image Classification under Distribution Shifts [28.21559601586271]
  HyperTTA (Test-Time Adaptable Transformer for Hyperspectral Degradation) is a unified framework that enhances model robustness under diverse degradation conditions.<n>Test-time adaptation strategy, the Confidence-aware Entropy-minimized LayerNorm Adapter (CELA), dynamically updates only the affine parameters of LayerNorm layers.<n>Experiments on two benchmark datasets demonstrate that HyperTTA outperforms state-of-the-art baselines across a wide range of degradation scenarios.
  arXiv  Detail & Related papers  (2025-09-10T09:31:37Z)
• A Principled Bayesian Framework for Training Binary and Spiking Neural Networks [1.6658912537684454]
  Spiking Bayesian Neural Networks (SBNNs) is a variational inference framework that uses posterior noise to train Binary and Spiking Neural Networks with IW-ST.<n>By linking low-bias conditions, vanishing gradients, and the KL term, we enable training of deep residual networks without normalisation.
  arXiv  Detail & Related papers  (2025-05-23T14:33:20Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• StyDeSty: Min-Max Stylization and Destylization for Single Domain Generalization [85.18995948334592]
  Single domain generalization (single DG) aims at learning a robust model generalizable to unseen domains from only one training domain. State-of-the-art approaches have mostly relied on data augmentations, such as adversarial perturbation and style enhancement, to synthesize new data. We propose emphStyDeSty, which explicitly accounts for the alignment of the source and pseudo domains in the process of data augmentation.
  arXiv  Detail & Related papers  (2024-06-01T02:41:34Z)
• DRAG: Divergence-based Adaptive Aggregation in Federated learning on Non-IID Data [11.830891255837788]
  Local gradient descent (SGD) is a fundamental approach in achieving communication efficiency in Federated Learning (FL) We introduce a novel metric called degree of divergence," quantifying the angle between the local gradient and the global reference direction. We propose the divergence-based adaptive aggregation (DRAG) algorithm, which dynamically drags" the received local updates toward the reference direction in each round without requiring extra communication overhead.
  arXiv  Detail & Related papers  (2023-09-04T19:40:58Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• Surgical Fine-Tuning Improves Adaptation to Distribution Shifts [114.17184775397067]
  A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model. This paper shows that in such settings, selectively fine-tuning a subset of layers matches or outperforms commonly used fine-tuning approaches.
  arXiv  Detail & Related papers  (2022-10-20T17:59:15Z)
• Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training [1.3999481573773072]
  We study the stability of these layers across runs and model sizes. We propose that group normalization may be used without disrupting their formation. We apply these findings to Federated Learning in order to improve the training procedure.
  arXiv  Detail & Related papers  (2021-10-08T17:25:19Z)
• Anatomy of Domain Shift Impact on U-Net Layers in MRI Segmentation [0.08156494881838945]
  SpotTUnet is a CNN architecture that automatically chooses the layers which should be optimally fine-tuned. We show that our method performs at the same level as the best of the nonflexible fine-tuning methods even under the extreme scarcity of annotated data.
  arXiv  Detail & Related papers  (2021-07-10T21:13:55Z)
• Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning [86.6929930921905]
  This paper studies how much it can help address domain shifts if we further have a few target samples labeled. To explore the full potential of landmarks, we incorporate a prototypical alignment (PA) module which calculates a target prototype for each class from the landmarks. Specifically, we severely perturb the labeled images, making PA non-trivial to achieve and thus promoting model generalizability.
  arXiv  Detail & Related papers  (2021-04-19T08:46:08Z)
• Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation [63.75774438196315]
  Unsupervised domain adaptation (UDA) aims to adapt existing models of the source domain to a new target domain with only unlabeled data. Most existing methods suffer from noticeable negative transfer resulting from either the error-prone discriminator network or the unreasonable teacher model. We propose an uncertainty-aware consistency regularization method for cross-domain semantic segmentation.
  arXiv  Detail & Related papers  (2020-04-19T15:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.