Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations

• URL: http://arxiv.org/abs/2409.08381v1
• Date: Thu, 12 Sep 2024 20:02:51 GMT
• Title: Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations
• Authors: Samyak Rawlekar, Shubhang Bhatnagar, Narendra Ahuja,
• Abstract summary: We study the impact of positive and negative prompt learning on Multi-Label Recognition (MLR) We introduce PositiveCoOp and NegativeCoOp, where only one prompt is learned with VLM guidance while the other is replaced by an embedding vector. We observe that negative prompts degrade MLR performance, and learning only positive prompts, combined with learned negative embeddings, outperforms dual prompt learning approaches.
• Score: 7.7149881834358345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) like CLIP have been adapted for Multi-Label Recognition (MLR) with partial annotations by leveraging prompt-learning, where positive and negative prompts are learned for each class to associate their embeddings with class presence or absence in the shared vision-text feature space. While this approach improves MLR performance by relying on VLM priors, we hypothesize that learning negative prompts may be suboptimal, as the datasets used to train VLMs lack image-caption pairs explicitly focusing on class absence. To analyze the impact of positive and negative prompt learning on MLR, we introduce PositiveCoOp and NegativeCoOp, where only one prompt is learned with VLM guidance while the other is replaced by an embedding vector learned directly in the shared feature space without relying on the text encoder. Through empirical analysis, we observe that negative prompts degrade MLR performance, and learning only positive prompts, combined with learned negative embeddings (PositiveCoOp), outperforms dual prompt learning approaches. Moreover, we quantify the performance benefits that prompt-learning offers over a simple vision-features-only baseline, observing that the baseline displays strong performance comparable to dual prompt learning approach (DualCoOp), when the proportion of missing labels is low, while requiring half the training compute and 16 times fewer parameters

Related papers

• Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models [33.39682202143465]
  Out-of-distribution (OOD) detection is committed to delineating the classification boundaries between in-distribution (ID) and OOD images.<n>Negative prompts are introduced to emphasize the dissimilarity between image features and prompt content.<n>We propose Positive and Negative Prompt Supervision, which encourages negative prompts to capture inter-class features.
  arXiv  Detail & Related papers  (2025-11-14T03:24:09Z)
• SL-SLR: Self-Supervised Representation Learning for Sign Language Recognition [0.5872014229110214]
  Sign language recognition ( SLR) is a machine learning task aiming to identify signs in videos.<n>In SLR, only certain parts provide information that is truly useful for its recognition.<n>This paper proposes a self-supervised learning framework designed to learn meaningful representations for SLR.
  arXiv  Detail & Related papers  (2025-09-05T15:38:19Z)
• Semantic-guided Representation Learning for Multi-Label Recognition [13.046479112800608]
  Multi-label Recognition (MLR) involves assigning multiple labels to each data instance in an image. Recent Vision and Language Pre-training methods have made significant progress in tackling zero-shot MLR tasks. We introduce a Semantic-guided Representation Learning approach (SigRL) that enables the model to learn effective visual and textual representations.
  arXiv  Detail & Related papers  (2025-04-04T08:15:08Z)
• Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation [22.057386630831402]
  Large Vision-Language Models can be instructed to solve diverse tasks by prompting, without task-specific training. We evaluate the segmentation performance of several recent models guided by either text or visual prompts. We propose PromptMatcher, a training-free baseline that combines both text and visual prompts.
  arXiv  Detail & Related papers  (2025-03-25T13:36:59Z)
• Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding [71.01099784480597]
  Large language models (LLMs) excel at a range of tasks through in-context learning (ICL) We introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping. ICCD emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples.
  arXiv  Detail & Related papers  (2025-02-19T14:04:46Z)
• Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model [27.56988000960972]
  We introduce a new framework based on a dual context of both domain-shared and class-specific contexts. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in Large Language Models. We also formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens.
  arXiv  Detail & Related papers  (2024-07-05T13:15:29Z)
• Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning [45.25602203155762]
  Self-Supervised Contrastive Learning has proven effective in deriving high-quality representations from unlabeled data. A major challenge that hinders both unimodal and multimodal contrastive learning is feature suppression. We propose a novel model-agnostic Multistage Contrastive Learning framework.
  arXiv  Detail & Related papers  (2024-02-19T04:13:33Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations [79.433122872973]
  Multi-label image recognition in the low-label regime is a task of great challenge and practical significance. We leverage the powerful alignment between textual and visual features pretrained with millions of auxiliary image-text pairs. We introduce an efficient and effective framework called Evidence-guided Dual Context Optimization (DualCoOp++)
  arXiv  Detail & Related papers  (2023-08-03T17:33:20Z)
• Reflection Invariance Learning for Few-shot Semantic Segmentation [53.20466630330429]
  Few-shot semantic segmentation (FSS) aims to segment objects of unseen classes in query images with only a few annotated support images. This paper proposes a fresh few-shot segmentation framework to mine the reflection invariance in a multi-view matching manner. Experiments on both PASCAL-$5textiti$ and COCO-$20textiti$ datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-01T15:14:58Z)
• Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractions [9.088460902782547]
  Clustering with Bisimulation Metrics (CBM) learns robust representations by grouping visual observations in the latent space. CBM alternates between two steps: (1) grouping observations by measuring their bisimulation distances to the learned prototypes; (2) learning a set of prototypes according to the current cluster assignments. Experiments demonstrate that CBM significantly improves the sample efficiency of popular visual RL algorithms.
  arXiv  Detail & Related papers  (2023-02-12T13:27:34Z)
• CPL: Counterfactual Prompt Learning for Vision and Language Models [76.18024920393245]
  This paper presents a novel underlinetextbfCounterfactual underlinetextbfPrompt underlinetextbfLearning (CPL) method for vision and language models. CPL simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks.
  arXiv  Detail & Related papers  (2022-10-19T08:06:39Z)
• DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations [61.41339201200135]
  We propose Dual Context Optimization (DualCoOp) as a unified framework for partial-label MLR and zero-shot MLR. Since DualCoOp only introduces a very light learnable overhead upon the pretrained vision-language framework, it can quickly adapt to multi-label recognition tasks.
  arXiv  Detail & Related papers  (2022-06-20T02:36:54Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Prototypical Contrastive Learning of Unsupervised Representations [171.3046900127166]
  Prototypical Contrastive Learning (PCL) is an unsupervised representation learning method. PCL implicitly encodes semantic structures of the data into the learned embedding space. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T09:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.