Progressive Conditioned Scale-Shift Recalibration of Self-Attention for Online Test-time Adaptation

• URL: http://arxiv.org/abs/2512.12673v1
• Date: Sun, 14 Dec 2025 12:56:02 GMT
• Title: Progressive Conditioned Scale-Shift Recalibration of Self-Attention for Online Test-time Adaptation
• Authors: Yushun Tang, Ziqiong Liu, Jiyuan Jia, Yi Zhang, Zhihai He,
• Abstract summary: Online test-time adaptation aims to dynamically adjust a network model in real-time based on sequential input samples during the inference stage.<n>We develop a new approach to progressively recalibrate the self-attention at each layer using a local linear transform parameterized by conditioned scale and shift factors.
• Score: 20.542420731967386
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online test-time adaptation aims to dynamically adjust a network model in real-time based on sequential input samples during the inference stage. In this work, we find that, when applying a transformer network model to a new target domain, the Query, Key, and Value features of its self-attention module often change significantly from those in the source domain, leading to substantial performance degradation of the transformer model. To address this important issue, we propose to develop a new approach to progressively recalibrate the self-attention at each layer using a local linear transform parameterized by conditioned scale and shift factors. We consider the online model adaptation from the source domain to the target domain as a progressive domain shift separation process. At each transformer network layer, we learn a Domain Separation Network to extract the domain shift feature, which is used to predict the scale and shift parameters for self-attention recalibration using a Factor Generator Network. These two lightweight networks are adapted online during inference. Experimental results on benchmark datasets demonstrate that the proposed progressive conditioned scale-shift recalibration (PCSR) method is able to significantly improve the online test-time domain adaptation performance by a large margin of up to 3.9\% in classification accuracy on the ImageNet-C dataset.

Related papers

• RemoteVAR: Autoregressive Visual Modeling for Remote Sensing Change Detection [52.32112533846212]
  Remote sensing change detection is central to applications such as environmental monitoring and disaster assessment.<n>Visual autoregressive models have recently shown impressive image generation capability, but their adoption for pixel-level discriminative tasks remains limited due to weak controllability, suboptimal dense prediction performance and exposure bias.<n>We introduce a new VAR-based change detection framework that addresses these limitations by conditioning autoregressive prediction on multi-resolution fused bi-temporal features via cross-attention, and by employing an autoregressive training strategy designed specifically for change map prediction.
  arXiv  Detail & Related papers  (2026-01-17T03:50:00Z)
• Simulating Distribution Dynamics: Liquid Temporal Feature Evolution for Single-Domain Generalized Object Detection [58.25418970608328]
  Single-Domain Generalized Object Detection (Single-DGOD) aims to transfer a detector trained on one source domain to multiple unknown domains.<n>Existing methods for Single-DGOD typically rely on discrete data augmentation or static perturbation methods to expand data diversity.<n>We propose a new method, which simulates the progressive evolution of features from the source domain to simulated latent distributions.
  arXiv  Detail & Related papers  (2025-11-13T03:10:39Z)
• GeneralizeFormer: Layer-Adaptive Model Generation across Test-Time Distribution Shifts [58.95913531746308]
  We consider the problem of test-time domain generalization, where a model is trained on several source domains and adjusted on target domains never seen during training.<n>We propose to generate multiple layer parameters on the fly during inference by a lightweight meta-learned transformer, which we call textitGeneralizeFormer.
  arXiv  Detail & Related papers  (2025-02-15T10:10:49Z)
• SelectiveFinetuning: Enhancing Transfer Learning in Sleep Staging through Selective Domain Alignment [3.5833494449195293]
  In practical sleep stage classification, a key challenge is the variability of EEG data across different subjects and environments.<n>Our method utilizes a pretrained Multi Resolution Convolutional Neural Network (MRCNN) to extract EEG features.<n>By finetuning the model with selective source data, our SelectiveFinetuning enhances the model's performance on target domain.
  arXiv  Detail & Related papers  (2025-01-07T13:08:54Z)
• Domain-Conditioned Transformer for Fully Test-time Adaptation [18.51468880167399]
  fully test-time adaptation aims to adapt a network model online based on sequential analysis of input samples during the inference stage. We observe that, when applying a transformer network model into a new domain, the self-attention profiles of image samples in the target domain deviate significantly from those in the source domain. We propose a new structure for the self-attention modules in the transformer. Specifically, we incorporate three domain-conditioning vectors, called domain conditioners, into the query, key, and value components of the self-attention module.
  arXiv  Detail & Related papers  (2024-10-14T12:36:27Z)
• Semantic Self-adaptation: Enhancing Generalization with a Single Sample [45.111358665370524]
  We propose a self-adaptive approach for semantic segmentation. It fine-tunes the parameters of convolutional layers to the input image using consistency regularization. Our empirical study suggests that self-adaptation may complement the established practice of model regularization at training time.
  arXiv  Detail & Related papers  (2022-08-10T12:29:01Z)
• Improving Test-Time Adaptation via Shift-agnostic Weight Regularization and Nearest Source Prototypes [18.140619966865955]
  We propose a novel test-time adaptation strategy that adjusts the model pre-trained on the source domain using only unlabeled online data from the target domain. We show that our method exhibits state-of-the-art performance on various standard benchmarks and even outperforms its supervised counterpart.
  arXiv  Detail & Related papers  (2022-07-24T10:17:05Z)
• Variational Transfer Learning using Cross-Domain Latent Modulation [1.9662978733004601]
  We introduce a novel cross-domain latent modulation mechanism to a variational autoencoder framework so as to achieve effective transfer learning. Deep representations of the source and target domains are first extracted by a unified inference model and aligned by employing gradient reversal. The learned deep representations are then cross-modulated to the latent encoding of the alternative domain, where consistency constraints are also applied.
  arXiv  Detail & Related papers  (2022-05-31T03:47:08Z)
• Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers [141.70707071815653]
  We propose a novel Sequence Feature Alignment (SFA) method that is specially designed for the adaptation of detection transformers. SFA consists of a domain query-based feature alignment (DQFA) module and a token-wise feature alignment (TDA) module. Experiments on three challenging benchmarks show that SFA outperforms state-of-the-art domain adaptive object detection methods.
  arXiv  Detail & Related papers  (2021-07-27T07:17:12Z)
• Transformer-Based Source-Free Domain Adaptation [134.67078085569017]
  We study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. We propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA.
  arXiv  Detail & Related papers  (2021-05-28T23:06:26Z)
• Cross-Domain Latent Modulation for Variational Transfer Learning [1.9212368803706577]
  We propose a cross-domain latent modulation mechanism within a variational autoencoders (VAE) framework to enable improved transfer learning. We apply the proposed model to a number of transfer learning tasks including unsupervised domain adaptation and image-to-image translation.
  arXiv  Detail & Related papers  (2020-12-21T22:45:00Z)
• On Robustness and Transferability of Convolutional Neural Networks [147.71743081671508]
  Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. We study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness.
  arXiv  Detail & Related papers  (2020-07-16T18:39:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.