Visual Distribution Anchoring for Efficient Prompt Tuning

• URL: http://arxiv.org/abs/2607.28967v1
• Date: Fri, 31 Jul 2026 02:43:07 GMT
• Title: Visual Distribution Anchoring for Efficient Prompt Tuning
• Abstract summary: We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework.<n>It augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool.<n>We show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect.
• Score: 0.6918455480131249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt tuning adapts vision--language models with few trainable parameters, but existing approaches trade off efficiency and adaptation: static textual prompts can overfit source classes, image-conditioned prompts add per-instance computation, and multimodal tuning modifies the visual branch. We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework that augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool. We first ask whether prototypes can be synthesized from class names. A text-to-centroid mapper reconstructs held-out source prototypes but fails under dataset shift because class names specify semantic identity, not target-domain appearance. An oracle analysis confirms that true target prototypes are highly discriminative. VDA therefore uses frozen semantic and domain-template classifiers to partition unlabeled target images into class-correlated groups. Confidence-ranked image features form normalized prototypes, fused with the semantic classifier using one global weight. Adaptation requires no target labels, target-side optimization, uniform class-prior assumption, iterative refinement, or test-query access, and yields a fixed, cacheable classifier. Controlled experiments show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect. Across ten ImageNet-to-target transfers, the same frozen design improves zero-shot CLIP, TCP, and MaPLe by 3.22, 3.39, and 3.35 points, respectively, improving nine of ten targets in every setting. Its visual correction further improves leakage-free PromptKD by 2.79 points, complementing zero-shot, source-prompted, multimodal-prompted, and target-distilled classifiers.

Related papers

• G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification [14.607727417863273]
  G2D is a training-free framework that uses a generative VLM to verify CLIP-retrieved candidates against the image.<n>It achieves 68.85% average accuracy, versus 59.35% for CLIP and 63.11% for the standalone VLM.<n>It also transfers to DCLIP, WaffleCLIP, and CuPL, supporting a practical interface between discriminative proposal and generative visual reasoning.
  arXiv  Detail & Related papers  (2026-08-27T07:34:07Z)
• Prompt-and-Transfer: Dynamic Class-aware Enhancement for Few-shot Segmentation [15.159690685421586]
  This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called Prompt and Transfer" (PAT) PAT constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS, Weak-label, and Zero-shot-label.
  arXiv  Detail & Related papers  (2024-09-16T15:24:26Z)
• DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models [7.649900082537232]
  This study introduces DPA, an unsupervised domain adaptation method for vision-textual models.<n>It introduces the concept of dual prototypes, acting as distinct classifiers, along with the convex combination of their outputs.<n>It ranks pseudo-labels to facilitate robust self-training, particularly during early training.<n>Experiments on 13 downstream vision tasks demonstrate that DPA significantly outperforms zero-shot CLIP and the state-of-the-art unsupervised adaptation baselines.
  arXiv  Detail & Related papers  (2024-08-16T17:30:27Z)
• Transductive Zero-Shot and Few-Shot CLIP [24.592841797020203]
  This paper addresses the transductive zero-shot and few-shot CLIP classification challenge. Inference is performed jointly across a mini-batch of unlabeled query samples, rather than treating each instance independently. Our approach yields near 20% improvement in ImageNet accuracy over CLIP's zero-shot performance.
  arXiv  Detail & Related papers  (2024-04-08T12:44:31Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation [20.57370550156505]
  ReCLIP is a source-free domain adaptation method for vision-language models. We demonstrate ReCLIP reduces the average error rate of CLIP from 30.17% to 25.06% on 22 image classification benchmarks.
  arXiv  Detail & Related papers  (2023-08-04T18:11:40Z)
• Diffusion Visual Counterfactual Explanations [51.077318228247925]
  Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image. Current approaches for the generation of VCEs are restricted to adversarially robust models and often contain non-realistic artefacts. In this paper, we overcome this by generating Visual Diffusion Counterfactual Explanations (DVCEs) for arbitrary ImageNet classifiers.
  arXiv  Detail & Related papers  (2022-10-21T09:35:47Z)
• Masked Unsupervised Self-training for Zero-shot Image Classification [98.23094305347709]
  Masked Unsupervised Self-Training (MUST) is a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification.
  arXiv  Detail & Related papers  (2022-06-07T02:03:06Z)
• Dual Prototypical Contrastive Learning for Few-shot Semantic Segmentation [55.339405417090084]
  We propose a dual prototypical contrastive learning approach tailored to the few-shot semantic segmentation (FSS) task. The main idea is to encourage the prototypes more discriminative by increasing inter-class distance while reducing intra-class distance in prototype feature space. We demonstrate that the proposed dual contrastive learning approach outperforms state-of-the-art FSS methods on PASCAL-5i and COCO-20i datasets.
  arXiv  Detail & Related papers  (2021-11-09T08:14:50Z)
• Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning [86.6929930921905]
  This paper studies how much it can help address domain shifts if we further have a few target samples labeled. To explore the full potential of landmarks, we incorporate a prototypical alignment (PA) module which calculates a target prototype for each class from the landmarks. Specifically, we severely perturb the labeled images, making PA non-trivial to achieve and thus promoting model generalizability.
  arXiv  Detail & Related papers  (2021-04-19T08:46:08Z)
• Fine-Grained Visual Classification with Efficient End-to-end Localization [49.9887676289364]
  We present an efficient localization module that can be fused with a classification network in an end-to-end setup. We evaluate the new model on the three benchmark datasets CUB200-2011, Stanford Cars and FGVC-Aircraft.
  arXiv  Detail & Related papers  (2020-05-11T14:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.