Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning

• URL: http://arxiv.org/abs/2605.29776v1
• Date: Thu, 28 May 2026 11:21:44 GMT
• Title: Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
• Abstract summary: This paper focuses on the target-domain few-shot finetuning in the CLIP-based CDFSL task.<n>We find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens improves target-domain performance.<n>Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP.
• Score: 25.20062959668559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) such as CLIP demonstrate strong zero-shot generalization, but their performance significantly degrades in cross-domain scenarios with scarce target-domain training data (Cross-Domain Few-Shot Learning, CDFSL). In this paper, we focus on the target-domain few-shot finetuning in the CLIP-based CDFSL task. Prevailing finetuning paradigms uniformly align all image patch tokens with their corresponding textual embeddings. However, we find a counterintuitive phenomenon: actively pushing away certain low-similarity image tokens, termed "tail tokens", from their textual embeddings consistently improves target-domain performance. We delve into this phenomenon and provide a novel interpretation: under great domain shifts and scarce training data, the model can hardly extract semantic information from visual inputs; therefore, the common belief of alignment is valid only for tokens already containing sufficient semantic information; for tail tokens, forcing the alignment would lead to excessive overfitting to the scarce training, while breaking the alignment is more useful. Motivated by this, we propose Adaptive Tail-Head Alignment (ATHA), a novel fine-tuning strategy for CLIP that transforms the conventional uniform alignment paradigm to an adaptive alignment paradigm, with both alignment strengthening and weakening. Extensive experiments on four challenging CDFSL benchmarks validate our state-of-the-art performance. Our code is available at https://github.com/shuaiyi308/ATHA.

Related papers

• Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning [25.20062959668559]
  Cross-Domain Few-Shot Learning can transfer source-domain information to target domains with scarce training data.<n>Standard target-domain few-shot fine-tuning exacerbates the attention sink problem, leading to poor discriminability across classes.<n>We propose a novel approach to dynamically re-weight tokens according to their relevance with target-domain classes during the target-domain finetuning.
  arXiv  Detail & Related papers  (2026-05-25T12:49:15Z)
• Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning [13.950318836859731]
  We propose SPA (Semantic-guided Patch-level Alignment) for CLIP-based CIL.<n>For each class, we first construct representative and diverse visual samples and feed them to GPT-5.<n>These descriptions are used to guide the selection of discriminative patch-level visual features.
  arXiv  Detail & Related papers  (2026-05-13T17:56:23Z)
• Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning [19.4465469479799]
  Cross-Domain Few-Shot Learning aims to adapt large-scale pretrained models to specialized target domains with limited samples.<n>We find adapter-based methods consistently outperform prompt-based ones, contrary to in-domain scenarios.<n>We propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods.
  arXiv  Detail & Related papers  (2026-05-12T07:20:23Z)
• Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models [89.0460992131069]
  We propose AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model.<n>Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss.<n>We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively.
  arXiv  Detail & Related papers  (2026-04-13T14:54:25Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• Multi-Prompt Progressive Alignment for Multi-Source Unsupervised Domain Adaptation [73.40696661117408]
  We propose a progressive alignment strategy for adapting CLIP to unlabeled downstream task.<n>We name our approach MP2A and test it on three popular UDA benchmarks, namely ImageCLEF, Office-Home, and the most challenging DomainNet.<n> Experiments showcase that MP2A achieves state-of-the-art performance when compared with most recent CLIP-based MS-UDA approaches.
  arXiv  Detail & Related papers  (2025-07-31T09:42:42Z)
• One Last Attention for Your Vision-Language Model [42.872184600248914]
  We propose textbfRational textbfAdaptaion (RAda) to explicitly exploit the final fused representation during fine-tuning.<n> RAda employs a learned mask, obtained from a lightweight attention layer attached at the end of a VLM, to dynamically calibrate the contribution of each element in the rational matrix.<n>Experiments show that RAda serves as a versatile fine-tuning technique, improving the baseline with minimal code and performing comparably against current arts in most settings.
  arXiv  Detail & Related papers  (2025-07-21T10:35:32Z)
• Locality Alignment Improves Vision-Language Models [55.275235524659905]
  Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors.<n>Our goal is to resolve this with a vision backbone that effectively captures both local and global image semantics.<n>We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed.
  arXiv  Detail & Related papers  (2024-10-14T21:01:01Z)
• Bootstrap Fine-Grained Vision-Language Alignment for Unified Zero-Shot Anomaly Localization [63.61093388441298]
  Contrastive Language-Image Pre-training models have shown promising performance on zero-shot visual recognition tasks. In this work, we propose AnoCLIP for zero-shot anomaly localization.
  arXiv  Detail & Related papers  (2023-08-30T10:35:36Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation [169.82760468633236]
  We propose to build the pixel-level cycle association between source and target pixel pairs. Our method can be trained end-to-end in one stage and introduces no additional parameters.
  arXiv  Detail & Related papers  (2020-10-31T00:11:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.