A$_3$B$_2$: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning

• URL: http://arxiv.org/abs/2605.13161v2
• Date: Fri, 15 May 2026 18:29:50 GMT
• Title: A$_3$B$_2$: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning
• Abstract summary: We propose an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning.<n>A$_3$B$$ introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image adaptation when prediction uncertainty is high.<n>Experiments on three few-shot image classification tasks across 11 datasets demonstrate that A$_3$B$ consistently outperforms competitive- and adapter-based baselines.
• Score: 24.50908296572901
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficient transfer learning methods for large-scale vision-language models ($e.g.$, CLIP) enable strong few-shot transfer, yet existing adaptation methods follow a fixed fine-tuning paradigm that implicitly assumes a uniform importance of the image and text branches, which has not been systematically studied in image classification. Through extensive analysis, we reveal a Branch Bias issue in vision-language image classification: adapting the image encoder does not always improve performance under out-of-distribution settings. Motivated by this observation, we propose A$_3$B$_2$, an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning. A$_3$B$_2$ introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image-branch adaptation when prediction uncertainty is high, enabling soft and data-driven control without manual intervention. Architecturally, A$_3$B$_2$ adopts a lightweight asymmetric design inspired by mixture-of-experts with Load Balancing Regularization. Extensive experiments on three few-shot image classification tasks across 11 datasets demonstrate that A$_3$B$_2$ consistently outperforms 11 competitive prompt- and adapter-based baselines.

Related papers

• HydraPrompt: An Adaptive and Asymmetric Framework of Vision-Language Models for Synthetic Image Detection [52.11418741192251]
  We propose an asymmetric prompting framework that adjusts the category centers by aligning with fine-grained image cues.<n>Specifically, we introduce a single set of prompts to capture the consistent representative patterns, which serves as a unified anchor for real content.<n>While (2) for fake category, we construct sample-adaptive prompts that specialize in capturing diverse cues from different samples, enabling adaptive modeling of forgery image variations.
  arXiv  Detail & Related papers  (2026-05-26T01:20:18Z)
• Uncertainty-Guided Selective Adaptation Enables Cross-Platform Predictive Fluorescence Microscopy [65.15943255667733]
  We introduce Subnetwork Image Translation ADDA with automatic depth selection (SIT-ADDA-Auto)<n>We show that adapting only the earliest convolutional layers, while freezing deeper layers, yields reliable transfer.<n>Our results provide a design rule for label-free adaptation in microscopy and a recipe for field settings; the code is publicly available.
  arXiv  Detail & Related papers  (2025-11-15T03:01:05Z)
• Training-Free Synthetic Data Generation with Dual IP-Adapter Guidance [13.893061390641348]
  DIPSY is a training-free approach to generate synthetic images using few-shot examples.<n>Our approach achieves state-of-the-art or comparable performance.<n>Our results highlight the effectiveness of leveraging dual image prompting with positive-negative guidance for generating class-discriminative features.
  arXiv  Detail & Related papers  (2025-09-26T17:57:32Z)
• Fine-grained Image Retrieval via Dual-Vision Adaptation [38.41312409147845]
  Fine-Grained Image Retrieval (FGIR) faces challenges in learning discriminative visual representations to retrieve images with similar fine-grained features.<n>We propose a Dual-Vision Adaptation (DVA) approach for FGIR, which guides the frozen pre-trained model to perform FGIR through collaborative sample and feature adaptation.
  arXiv  Detail & Related papers  (2025-06-19T12:46:55Z)
• CLIP Adaptation by Intra-modal Overlap Reduction [1.2277343096128712]
  We analyse the intra-modal overlap in image space in terms of embedding representation. We train a lightweight adapter on a generic set of samples from the Google Open Images dataset.
  arXiv  Detail & Related papers  (2024-09-17T16:40:58Z)
• Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval [92.13664084464514]
  The task of composed image retrieval (CIR) aims to retrieve images based on the query image and the text describing the users' intent. Existing methods have made great progress with the advanced large vision-language (VL) model in CIR task, however, they generally suffer from two main issues: lack of labeled triplets for model training and difficulty of deployment on resource-restricted environments. We propose Image2Sentence based Asymmetric zero-shot composed image retrieval (ISA), which takes advantage of the VL model and only relies on unlabeled images for composition learning.
  arXiv  Detail & Related papers  (2024-03-03T07:58:03Z)
• Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detection [106.39544368711427]
  We study the problem of generalizable synthetic image detection, aiming to detect forgery images from diverse generative methods. We present a novel forgery-aware adaptive transformer approach, namely FatFormer. Our approach tuned on 4-class ProGAN data attains an average of 98% accuracy to unseen GANs, and surprisingly generalizes to unseen diffusion models with 95% accuracy.
  arXiv  Detail & Related papers  (2023-12-27T17:36:32Z)
• Class Incremental Learning with Pre-trained Vision-Language Models [59.15538370859431]
  We propose an approach to exploiting pre-trained vision-language models (e.g. CLIP) that enables further adaptation. Experiments on several conventional benchmarks consistently show a significant margin of improvement over the current state-of-the-art.
  arXiv  Detail & Related papers  (2023-10-31T10:45:03Z)
• Train/Test-Time Adaptation with Retrieval [129.8579208970529]
  We introduce Train/Test-Time Adaptation with Retrieval ($rm T3AR$), a method to adapt models both at train and test time. $rm T3AR$ adapts a given model to the downstream task using refined pseudo-labels and a self-supervised contrastive objective function. Thanks to the retrieval module, our method gives the user or service provider the possibility to improve model adaptation on the downstream task.
  arXiv  Detail & Related papers  (2023-03-25T02:44:57Z)
• Rethinking Generalization in Few-Shot Classification [28.809141478504532]
  Single image-level annotations only correctly describe an often small subset of an image's content. In this paper, we take a closer look at the implications in the context of $textitfew-shot learning$. We build on recent advances in unsupervised training of networks via masked image modelling to overcome the lack of fine-grained labels.
  arXiv  Detail & Related papers  (2022-06-15T03:05:21Z)
• Self-supervised Augmentation Consistency for Adapting Semantic Segmentation [56.91850268635183]
  We propose an approach to domain adaptation for semantic segmentation that is both practical and highly accurate. We employ standard data augmentation techniques $-$ photometric noise, flipping and scaling $-$ and ensure consistency of the semantic predictions. We achieve significant improvements of the state-of-the-art segmentation accuracy after adaptation, consistent both across different choices of the backbone architecture and adaptation scenarios.
  arXiv  Detail & Related papers  (2021-04-30T21:32:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.