Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

• URL: http://arxiv.org/abs/2607.18695v1
• Date: Tue, 21 Jul 2026 04:34:24 GMT
• Title: Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models
• Abstract summary: A popular route to interpretable zero-shot classification asks a large language model to describe each class name and prompts CLIP with the resulting descriptors.<n>We show that these descriptors carry little visual evidence of their own.<n>We select attributes from the target image collection instead.
• Score: 67.53510678294136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors. We show that these descriptors carry little visual evidence of their own: removing the class name from the prompt collapses ImageNet accuracy from 59.5% to 15.5%. The diagnosis is that the descriptors are conditioned on the label rather than on the images, so they describe the concept in general and mislead exactly when the data shifts; an LLM insists that strawberries are red, but every strawberry in ImageNet-Sketch is a colorless line drawing. We therefore select attributes from the target image collection instead: we score a large attribute pool against the images in CLIP's joint embedding space and keep the top-scoring attributes per class. Selected this way, class-name-free attribute prompts reach 23.8% on ImageNet (against 15.5% for LLM descriptors), the gain holds on four shifted ImageNet variants, and reselecting from the LLM's own pool isolates the selection mechanism as the cause. With one image per class, the selected attributes outperform the prompt-tuning method CoOp by 3 points while fitting in under a minute instead of 14 hours, with no learned soft prompt to obscure the decision. Because the attribute set is chosen by the data, it doubles as a readable summary of a dataset, which we use to describe distribution shift in words.

Related papers

• Visual Distribution Anchoring for Efficient Prompt Tuning [0.6918455480131249]
  We propose VDA (Visual Distribution Anchoring), a training-free target adaptation framework.<n>It augments a frozen semantic classifier with class-level visual prototypes estimated offline from an unlabeled target pool.<n>We show that class-specific partitioning drives gains and that visually local pseudo-label errors can remain useful despite being class-incorrect.
  arXiv  Detail & Related papers  (2026-07-31T02:43:07Z)
• CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models [63.267844001341814]
  We propose class-aware zero-shot prompt reweighting (CARPRT)<n>We quantify class-specific relevance by averaging image-text relevance scores over images predicted to that class under the given prompt.<n> Evaluations on standard image classification benchmarks show that CARPRT outperforms existing class-independent reweighting methods.
  arXiv  Detail & Related papers  (2026-06-19T07:15:46Z)
• Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs [44.21486904657393]
  AutoSEP is a self-supervised prompt learning framework designed to enhance fine-grained classification capabilities.<n>Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features.<n>AutoSEP on average improves 13 percent over standard zero-shot classification and 5 percent over the best-performing baselines.
  arXiv  Detail & Related papers  (2025-06-01T09:04:07Z)
• Does VLM Classification Benefit from LLM Description Semantics? [26.743684911323857]
  We propose a training-free method for selecting discriminative descriptions that work independently of classname-ensembling effects.<n>Our approach identifies descriptions that effectively differentiate classes within a local CLIP label neighborhood, improving classification accuracy across seven datasets.
  arXiv  Detail & Related papers  (2024-12-16T16:01:18Z)
• Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classification [8.663915072332834]
  Follow-up Differential Descriptions (FuDD) is a zero-shot approach that tailors the class descriptions to each dataset. FuDD first identifies the ambiguous classes for each image, and then uses a Large Language Model (LLM) to generate new class descriptions that differentiate between them. We show that FuDD consistently outperforms generic description ensembles and naive LLM-generated descriptions on 12 datasets.
  arXiv  Detail & Related papers  (2023-11-10T05:24:07Z)
• Text Descriptions are Compressive and Invariant Representations for Visual Learning [63.3464863723631]
  We show that an alternative approach, in line with humans' understanding of multiple visual features per class, can provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, textit SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify
  arXiv  Detail & Related papers  (2023-07-10T03:06:45Z)
• Waffling around for Performance: Visual Classification with Random Words and Broad Concepts [121.60918966567657]
  WaffleCLIP is a framework for zero-shot visual classification which simply replaces LLM-generated descriptors with random character and word descriptors. We conduct an extensive experimental study on the impact and shortcomings of additional semantics introduced with LLM-generated descriptors.
  arXiv  Detail & Related papers  (2023-06-12T17:59:48Z)
• Improving Zero-shot Generalization and Robustness of Multi-modal Models [70.14692320804178]
  Multi-modal image-text models such as CLIP and LiT have demonstrated impressive performance on image classification benchmarks. We investigate the reasons for this performance gap and find that many of the failure cases are caused by ambiguity in the text prompts. We propose a simple and efficient way to improve accuracy on such uncertain images by making use of the WordNet hierarchy.
  arXiv  Detail & Related papers  (2022-12-04T07:26:24Z)
• What does a platypus look like? Generating customized prompts for zero-shot image classification [52.92839995002636]
  This work introduces a simple method to generate higher accuracy prompts without relying on any explicit knowledge of the task domain. We leverage the knowledge contained in large language models (LLMs) to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet.
  arXiv  Detail & Related papers  (2022-09-07T17:27:08Z)
• Attribute Group Editing for Reliable Few-shot Image Generation [85.52840521454411]
  We propose a new editing-based method, i.e., Attribute Group Editing (AGE), for few-shot image generation. AGE examines the internal representation learned in GANs and identifies semantically meaningful directions.
  arXiv  Detail & Related papers  (2022-03-16T06:54:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.