Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning

• URL: http://arxiv.org/abs/2605.25799v1
• Date: Mon, 25 May 2026 12:49:15 GMT
• Title: Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning
• Abstract summary: Cross-Domain Few-Shot Learning can transfer source-domain information to target domains with scarce training data.<n>Standard target-domain few-shot fine-tuning exacerbates the attention sink problem, leading to poor discriminability across classes.<n>We propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning.
• Score: 25.20062959668559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) like CLIP have shown impressive generalization capabilities, yet their potential for Cross-Domain Few-Shot Learning (CDFSL) remains underexplored, where the model needs to transfer source-domain information to target domains with scarce training data. While the attention sink phenomenon has been observed in VLMs for certain tasks, its role in CDFSL scenarios has not been studied. In this paper, we uncover a critical issue overlooked by prior works: standard target-domain few-shot fine-tuning in CDFSL significantly exacerbates the attention sink problem, leading to poor discriminability across classes. To understand this phenomenon, through extensive experiments, we interpret it as the model's shortcut learning for domain adaptation: to overcome the huge domain gap between the source and target domains, the model shows a high tendency to push tokens that are initially closer to target-domain classes (i.e., simple tokens) to be even closer to these classes, exacerbating the attention sink and wasting the capability of learning other discriminative but initially further tokens (i.e., hard tokens). To address this, we propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning, which explicitly suppresses the model's reliance on these simple tokens and enhances the learning of hard tokens, reducing sink tokens and enhancing discriminability. Extensive experiments on four benchmark datasets validate the rationale of our method, demonstrating new state-of-the-art performance. Our codes are available at https://github.com/shuaiyi308/TIR.

Related papers

• Random Registers for Cross-Domain Few-Shot Learning [19.199947811410123]
  Cross-domain few-shot learning aims to transfer knowledge from a data-sufficient source domain to data-scarce target domains.<n>We find that during the source-domain training, prompt tuning, as a common way to train ViT, could be harmful for the generalization of ViT in target domains.<n>We propose a simple but effective approach for CDFSL by adding random registers on the semantic regions of image tokens.
  arXiv  Detail & Related papers  (2025-06-03T13:13:58Z)
• CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning [52.63674911541416]
  Few-shot class-incremental learning (FSCIL) faces several challenges, such as overfitting and forgetting. Our primary focus is representation learning on base classes to tackle the unique challenge of FSCIL. We find that trying to secure the spread of features within a more confined feature space enables the learned representation to strike a better balance between transferability and discriminability.
  arXiv  Detail & Related papers  (2024-10-08T02:23:16Z)
• Learning to Discover Knowledge: A Weakly-Supervised Partial Domain Adaptation Approach [20.899013563493202]
  Domain adaptation has shown appealing performance by leveraging knowledge from a source domain with rich annotations. For a specific target task, it is cumbersome to collect related and high-quality source domains. In this paper, we propose a simple yet effective domain adaptation approach, termed as self-paced transfer classifier learning (SP-TCL)
  arXiv  Detail & Related papers  (2024-06-20T12:54:07Z)
• CDFSL-V: Cross-Domain Few-Shot Learning for Videos [58.37446811360741]
  Few-shot video action recognition is an effective approach to recognizing new categories with only a few labeled examples. Existing methods in video action recognition rely on large labeled datasets from the same domain. We propose a novel cross-domain few-shot video action recognition method that leverages self-supervised learning and curriculum learning.
  arXiv  Detail & Related papers  (2023-09-07T19:44:27Z)
• Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentation [80.1412989006262]
  Domain adaptive semantic segmentation aims to transfer knowledge from a labeled source domain to an unlabeled target domain. We propose T2S-DA, which we interpret as a form of pulling Target to Source for Domain Adaptation.
  arXiv  Detail & Related papers  (2023-05-23T07:09:09Z)
• Deep Learning for Cross-Domain Few-Shot Visual Recognition: A Survey [33.00835033658241]
  Few-shot learning enables models to perform the target tasks with very few labeled examples.<n>To overcome this limitation, Cross-domain few-shot learning has gained attention.<n>This paper presents the first comprehensive review of Cross-domain Few-shot Learning.
  arXiv  Detail & Related papers  (2023-03-15T12:18:16Z)
• Self-training through Classifier Disagreement for Cross-Domain Opinion Target Extraction [62.41511766918932]
  Opinion target extraction (OTE) or aspect extraction (AE) is a fundamental task in opinion mining. Recent work focus on cross-domain OTE, which is typically encountered in real-world scenarios. We propose a new SSL approach that opts for selecting target samples whose model output from a domain-specific teacher and student network disagrees on the unlabelled target data.
  arXiv  Detail & Related papers  (2023-02-28T16:31:17Z)
• Deep face recognition with clustering based domain adaptation [57.29464116557734]
  We propose a new clustering-based domain adaptation method designed for face recognition task in which the source and target domain do not share any classes. Our method effectively learns the discriminative target feature by aligning the feature domain globally, and, at the meantime, distinguishing the target clusters locally.
  arXiv  Detail & Related papers  (2022-05-27T12:29:11Z)
• Unsupervised Transfer Learning with Self-Supervised Remedy [60.315835711438936]
  Generalising deep networks to novel domains without manual labels is challenging to deep learning. Pre-learned knowledge does not transfer well without making strong assumptions about the learned and the novel domains. In this work, we aim to learn a discriminative latent space of the unlabelled target data in a novel domain by knowledge transfer from labelled related domains.
  arXiv  Detail & Related papers  (2020-06-08T16:42:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.