Learning Generalizable Prompt for CLIP with Class Similarity Knowledge

• URL: http://arxiv.org/abs/2502.11969v1
• Date: Mon, 17 Feb 2025 16:18:07 GMT
• Title: Learning Generalizable Prompt for CLIP with Class Similarity Knowledge
• Abstract summary: We propose Similarity Alignment Regularization (SAR) to preserve semantic relationships among classes captured by hand-crafted prompts.<n>We first obtain novel classes related to base classes using ChatGPT-4o and utilize them as potential unseen classes during prompt tuning.<n>Then, by targeting both base and novel classes, SAR aligns the similarity relationships among text embeddings generated by learnable prompts with the similarity relationships from hand-crafted prompts.
• Score: 1.3198143828338362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In vision-language models (VLMs), prompt tuning has shown its effectiveness in adapting models to downstream tasks. However, learned prompts struggle to generalize to unseen classes, as they tend to overfit to the classes that are targeted during prompt tuning. Examining failure cases, we observed that learned prompts disrupt the semantics of unseen classes, generating text embeddings with incorrect semantic relationships among classes. To address this, we propose Similarity Alignment Regularization (SAR), which regularizes learnable prompts to preserve the semantic relationships among classes captured by hand-crafted prompts. Specifically, we first obtain novel classes related to base classes using ChatGPT-4o and utilize them as potential unseen classes during prompt tuning. Then, by targeting both base and novel classes, SAR aligns the similarity relationships among text embeddings generated by learnable prompts with the similarity relationships from hand-crafted prompts. Extensive experiments applying SAR to existing prompt tuning methods demonstrate its effectiveness in improving generalization to unseen classes.

Related papers

• Decoupling Augmentation Bias in Prompt Learning for Vision-Language Models [8.634414503821697]
  Methods such as CoCoOp have shown that replacing handcrafted prompts with learnable vectors, known as prompt learning, can result in improved performance.<n>While traditional zero-shot learning techniques benefit from various data augmentation strategies, prompt learning has primarily focused on text-based modifications.<n>We explore how image-level augmentations, particularly those that introduce attribute-specific variations, can support and enhance prompt learning.
  arXiv  Detail & Related papers  (2025-11-05T11:15:16Z)
• SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting [70.49268117587562]
  We propose a plug-and-play Semantic-Driven Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the training set to unseen categories. During inference, we dynamically synthesize the visual prompts for unseen categories based on the semantic correlation between unseen and training categories.
  arXiv  Detail & Related papers  (2025-04-24T09:31:08Z)
• CAARMA: Class Augmentation with Adversarial Mixup Regularization [34.02819618734268]
  CAARMA is a class augmentation framework for speaker verification. We introduce synthetic classes through data mixing in the embedding space, expanding the number of training classes. We evaluate CAARMA on multiple speaker verification tasks, as well as other representative zero-shot comparison-based speech analysis tasks.
  arXiv  Detail & Related papers  (2025-03-20T21:41:16Z)
• A Similarity Paradigm Through Textual Regularization Without Forgetting [17.251684463032433]
  We propose a novel method called Similarity Paradigm with Textual Regularization (SPTR) for prompt learning without forgetting. SPTR is a two-pronged design based on hand-crafted prompts that is an inseparable framework. Four representative tasks across 11 datasets demonstrate that SPTR outperforms existing prompt learning methods.
  arXiv  Detail & Related papers  (2025-02-20T09:06:44Z)
• DesCLIP: Robust Continual Adaptation via General Attribute Descriptions for Pretrained Vision-Language Models [13.917530818500481]
  Continual adaptation of vision-language models (VLMs) focuses on leveraging cross-modal pretrained knowledge to incrementally adapt for expanding downstream tasks and datasets.<n>Existing research often focuses on connecting visual features with specific class text in downstream tasks, overlooking the latent relationships between general and specialized knowledge.<n>We propose DesCLIP, which leverages general attribute (GA) descriptions to guide the understanding of specific class objects.
  arXiv  Detail & Related papers  (2025-02-02T01:06:02Z)
• Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP [24.22470408549266]
  We dub prompt embedding as Aggregate-and-Adapted Prompt Embedding (AAPE) AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language understanding tasks. We also show AAPE is particularly helpful to handle non-canonical and OOD examples.
  arXiv  Detail & Related papers  (2024-10-31T07:41:13Z)
• TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model [78.77544632773404]
  We present a Textual-based Class-aware Prompt tuning( TCP) that explicitly incorporates prior knowledge about classes to enhance their discriminability. TCP consistently achieves superior performance while demanding less training time.
  arXiv  Detail & Related papers  (2023-11-30T03:59:23Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• LabelPrompt: Effective Prompt-based Learning for Relation Classification [31.291466190218912]
  This paper presents a novel prompt-based learning method, namely LabelPrompt, for the relation classification task. Motivated by the intuition to GIVE MODEL CHOICES!'', we first define additional tokens to represent relation labels, which regard these tokens as the verbaliser with semantic initialisation. Then, to mitigate inconsistency between predicted relations and given entities, we implement an entity-aware module with contrastive learning.
  arXiv  Detail & Related papers  (2023-02-16T04:06:25Z)
• Generative Prompt Tuning for Relation Classification [21.027631157115135]
  We propose a novel generative prompt tuning method to reformulate relation classification as an infilling problem. In addition, we design entity-guided decoding and discriminative relation scoring to generate and align relations effectively and efficiently during inference.
  arXiv  Detail & Related papers  (2022-10-22T12:40:23Z)
• LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models [67.19124099815645]
  We propose a novel Language-Aware Soft Prompting (LASP) learning method to alleviate base class overfitting. LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available. LASP matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets.
  arXiv  Detail & Related papers  (2022-10-03T17:56:35Z)
• MGIMN: Multi-Grained Interactive Matching Network for Few-shot Text Classification [9.9875634964736]
  Text classification struggles to generalize to unseen classes with very few labeled text instances per class. We propose a meta-learning based method MGIMN which performs instance-wise comparison followed by aggregation to generate class-wise matching vectors.
  arXiv  Detail & Related papers  (2022-04-11T08:58:55Z)
• Open-Set Representation Learning through Combinatorial Embedding [62.05670732352456]
  We are interested in identifying novel concepts in a dataset through representation learning based on the examples in both labeled and unlabeled classes. We propose a learning approach, which naturally clusters examples in unseen classes using the compositional knowledge given by multiple supervised meta-classifiers on heterogeneous label spaces. The proposed algorithm discovers novel concepts via a joint optimization of enhancing the discrimitiveness of unseen classes as well as learning the representations of known classes generalizable to novel ones.
  arXiv  Detail & Related papers  (2021-06-29T11:51:57Z)
• PTR: Prompt Tuning with Rules for Text Classification [64.1655047016891]
  Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. We propose prompt tuning with rules (PTR) for many-class text classification. PTR is able to encode prior knowledge of each class into prompt tuning.
  arXiv  Detail & Related papers  (2021-05-24T13:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.