CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models

• URL: http://arxiv.org/abs/2607.14125v1
• Date: Fri, 19 Jun 2026 07:15:46 GMT
• Title: CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models
• Abstract summary: We propose class-aware zero-shot prompt reweighting (CARPRT)<n>We quantify class-specific relevance by averaging image-text relevance scores over images predicted to that class under the given prompt.<n> Evaluations on standard image classification benchmarks show that CARPRT outperforms existing class-independent reweighting methods.
• Score: 63.267844001341814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained vision-language models (VLMs) enable zero-shot image classification by computing the similarity score between an image and textual descriptions, typically formed by inserting a class label (e.g., "cat") into a prompt (e.g., "a photo of a"). Since the score for a given image-class pair is sensitive to the choice of prompt, existing studies ensemble multiple prompts using a weighting vector to aggregate scores across different prompts. Yet, in current strategies, the weighting vector assigned to each prompt is shared across all classes, implicitly assuming that prompts are conditionally independent of classes, which often does not hold in practice, as a prompt like "an aerial view of" might be apt for "airport" but ill-suited for "apple". To address this, we propose class-aware zero-shot prompt reweighting (CARPRT). This scoring scheme adjusts the weighting vector for each class label by capturing the class-specific relevance of different prompts in a training-free manner. For each class label and every available prompt, we quantify their class-specific relevance by averaging image-text relevance scores over images predicted to that class under the given prompt. These estimates are then normalized to derive class-specific weights. Evaluations on standard image classification benchmarks show that CARPRT outperforms existing class-independent reweighting methods, confirming that modeling prompt-class dependencies is crucial for effective zero-shot prediction and even broader VLM-based application settings that rely on prompt ensembling. Our code is available at https://github.com/tmlr-group/CARPRT.

Related papers

• Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models [67.53510678294136]
  A popular route to interpretable zero-shot classification asks a large language model to describe each class name and prompts CLIP with the resulting descriptors.<n>We show that these descriptors carry little visual evidence of their own.<n>We select attributes from the target image collection instead.
  arXiv  Detail & Related papers  (2026-07-21T04:34:24Z)
• Classifier Enhancement Using Extended Context and Domain Experts for Semantic Segmentation [16.76156233696562]
  We propose an Extended Context-Aware (ECAC) that dynamically adjusts the classifier using global (dataset-level) and local (image-level) contextual information.<n>ECAC can achieve state-of-the-art performance across several datasets, including ADE20K, COCO-Stuff10K, and Pascal-Context.
  arXiv  Detail & Related papers  (2025-10-29T05:17:13Z)
• Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptation [40.60703048681749]
  Vision-language models (VLMs) such as CLIP achieve zero-shot transfer across various tasks by pre-training on numerous image-text pairs.<n>Despite being effective, conventional prompt ensembling that averages textual features of context prompts often yields suboptimal results.<n>We propose the Cluster-Aware Prompt Ensemble Learning framework, which preserves the cluster nature of context prompts.
  arXiv  Detail & Related papers  (2025-10-10T20:58:43Z)
• SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting [70.49268117587562]
  We propose a plug-and-play Semantic-Driven Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the training set to unseen categories.<n>During inference, we dynamically synthesize the visual prompts for unseen categories based on the semantic correlation between unseen and training categories.
  arXiv  Detail & Related papers  (2025-04-24T09:31:08Z)
• A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis [24.531871371861477]
  We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR) We name our approach INterpretable TRansformer (INTR) which is fairly easy to implement and exhibits several compelling properties. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site.
  arXiv  Detail & Related papers  (2023-11-07T17:32:55Z)
• AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models [12.971616352162435]
  AutoCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similarities. We show that AutoCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.
  arXiv  Detail & Related papers  (2023-09-28T13:08:08Z)
• Text Descriptions are Compressive and Invariant Representations for Visual Learning [63.3464863723631]
  We show that an alternative approach, in line with humans' understanding of multiple visual features per class, can provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, textit SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify
  arXiv  Detail & Related papers  (2023-07-10T03:06:45Z)
• Visual Classification via Description from Large Language Models [23.932495654407425]
  Vision-language models (VLMs) have shown promising performance on a variety of recognition tasks. We present an alternative framework for classification with VLMs, which we call classification by description.
  arXiv  Detail & Related papers  (2022-10-13T17:03:46Z)
• What does a platypus look like? Generating customized prompts for zero-shot image classification [52.92839995002636]
  This work introduces a simple method to generate higher accuracy prompts without relying on any explicit knowledge of the task domain. We leverage the knowledge contained in large language models (LLMs) to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet.
  arXiv  Detail & Related papers  (2022-09-07T17:27:08Z)
• Prompt-aligned Gradient for Prompt Tuning [59.04562893153831]
  We present Prompt-aligned Gradient, dubbed ProGrad, to prevent prompt tuning from forgetting the general knowledge learned from vision-language models (VLMs)<n>ProGrad only updates the prompt whose gradient is aligned to the "general direction", which is represented as the gradient of the KL loss of the pre-defined prompt prediction.<n>Experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods.
  arXiv  Detail & Related papers  (2022-05-30T06:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.