AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models

• URL: http://arxiv.org/abs/2309.16414v3
• Date: Wed, 14 Aug 2024 09:06:00 GMT
• Title: AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
• Abstract summary: AutoCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similarities. We show that AutoCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.
• Score: 12.971616352162435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Classifiers built upon vision-language models such as CLIP have shown remarkable zero-shot performance across a broad range of image classification tasks. Prior work has studied different ways of automatically creating descriptor sets for every class based on prompt templates, ranging from manually engineered templates over templates obtained from a large language model to templates built from random words and characters. Up until now, deriving zero-shot classifiers from the respective encoded class descriptors has remained nearly unchanged, i.e., classify to the class that maximizes cosine similarity between its averaged encoded class descriptors and the image encoding. However, weighing all class descriptors equally can be suboptimal when certain descriptors match visual clues on a given image better than others. In this work, we propose AutoCLIP, a method for auto-tuning zero-shot classifiers. AutoCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similarities. AutoCLIP is fully unsupervised, has only a minor additional computation overhead, and can be easily implemented in few lines of code. We show that AutoCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.

Related papers

• CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models [63.267844001341814]
  We propose class-aware zero-shot prompt reweighting (CARPRT)<n>We quantify class-specific relevance by averaging image-text relevance scores over images predicted to that class under the given prompt.<n> Evaluations on standard image classification benchmarks show that CARPRT outperforms existing class-independent reweighting methods.
  arXiv  Detail & Related papers  (2026-06-19T07:15:46Z)
• Can Synthetic Images Serve as Effective and Efficient Class Prototypes? [4.813908624670794]
  Contrastive Language-Image Pre-training (CLIP) relies on annotated text-to-image pairs for aligning visual and textual modalities.<n>This dependency introduces substantial cost and accuracy requirement in preparing high-quality datasets.<n>We introduce a Contrastive Language-Image Pre-training via Large-Language-Model-based Generation (LGCLIP)" framework.
  arXiv  Detail & Related papers  (2025-12-19T01:39:43Z)
• Hierarchical Representation Matching for CLIP-based Class-Incremental Learning [80.2317078787969]
  Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams.<n>Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task.<n>We introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL.
  arXiv  Detail & Related papers  (2025-09-26T17:59:51Z)
• Making Better Mistakes in CLIP-Based Zero-Shot Classification with Hierarchy-Aware Language Prompts [2.967024581564439]
  We introduce making better mistakes in CLIP-based zero-shot classification. Our approach outperforms the related methods in a holistic comparison across five datasets.
  arXiv  Detail & Related papers  (2025-03-04T03:54:50Z)
• CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collections [22.32157080294386]
  We propose a label-free prompt-tuning method to enhance CLIP-based image classification performance using unlabeled images. Our framework, NoLA (No Labels Attached), achieves an average absolute gain of 3.6% over the state-of-the-art LaFTer.
  arXiv  Detail & Related papers  (2024-11-28T19:48:54Z)
• CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement [65.47237619200442]
  Contrastive language image pretraining (CLIP) is a standard method for training vision-language models. We augment CLIP training with task-specific vision models from model zoos to improve its visual representations. This simple setup shows substantial improvements of up to 16.3% across different vision tasks.
  arXiv  Detail & Related papers  (2023-10-21T20:20:13Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)
• Text Descriptions are Compressive and Invariant Representations for Visual Learning [63.3464863723631]
  We show that an alternative approach, in line with humans' understanding of multiple visual features per class, can provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, textit SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify
  arXiv  Detail & Related papers  (2023-07-10T03:06:45Z)
• Multi-Modal Classifiers for Open-Vocabulary Object Detection [104.77331131447541]
  The goal of this paper is open-vocabulary object detection (OVOD) We adopt a standard two-stage object detector architecture. We explore three ways via: language descriptions, image exemplars, or a combination of the two.
  arXiv  Detail & Related papers  (2023-06-08T18:31:56Z)
• I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification [108.83932812826521]
  Large Language Models (LLM) trained on web-scale text show impressive abilities to repurpose their learned knowledge for a multitude of tasks. Our proposed model, I2MVFormer, learns multi-view semantic embeddings for zero-shot image classification with these class views. I2MVFormer establishes a new state-of-the-art on three public benchmark datasets for zero-shot image classification with unsupervised semantic embeddings.
  arXiv  Detail & Related papers  (2022-12-05T14:11:36Z)
• Text2Model: Text-based Model Induction for Zero-shot Image Classification [38.704831945753284]
  We address the challenge of building task-agnostic classifiers using only text descriptions. We generate zero-shot classifiers using a hypernetwork that receives class descriptions and outputs a multi-class model. We evaluate this approach in a series of zero-shot classification tasks, for image, point-cloud, and action recognition, using a range of text descriptions.
  arXiv  Detail & Related papers  (2022-10-27T05:19:55Z)
• Visual Classification via Description from Large Language Models [23.932495654407425]
  Vision-language models (VLMs) have shown promising performance on a variety of recognition tasks. We present an alternative framework for classification with VLMs, which we call classification by description.
  arXiv  Detail & Related papers  (2022-10-13T17:03:46Z)
• What does a platypus look like? Generating customized prompts for zero-shot image classification [52.92839995002636]
  This work introduces a simple method to generate higher accuracy prompts without relying on any explicit knowledge of the task domain. We leverage the knowledge contained in large language models (LLMs) to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet.
  arXiv  Detail & Related papers  (2022-09-07T17:27:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.