Visual-Semantic Contrastive Alignment for Few-Shot Image Classification

• URL: http://arxiv.org/abs/2210.11000v1
• Date: Thu, 20 Oct 2022 03:59:40 GMT
• Title: Visual-Semantic Contrastive Alignment for Few-Shot Image Classification
• Authors: Mohamed Afham, Ranga Rodrigo
• Abstract summary: Few-Shot learning aims to train a model that can adapt to unseen visual classes with only a few labeled examples. We introduce a contrastive alignment mechanism for visual and semantic feature vectors to learn much more generalized visual concepts. Our method simply adds an auxiliary contrastive learning objective which captures the contextual knowledge of a visual category.
• Score: 1.109560166867076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-Shot learning aims to train and optimize a model that can adapt to unseen visual classes with only a few labeled examples. The existing few-shot learning (FSL) methods, heavily rely only on visual data, thus fail to capture the semantic attributes to learn a more generalized version of the visual concept from very few examples. However, it is a known fact that human visual learning benefits immensely from inputs from multiple modalities such as vision, language, and audio. Inspired by the human learning nature of encapsulating the existing knowledge of a visual category which is in the form of language, we introduce a contrastive alignment mechanism for visual and semantic feature vectors to learn much more generalized visual concepts for few-shot learning. Our method simply adds an auxiliary contrastive learning objective which captures the contextual knowledge of a visual category from a strong textual encoder in addition to the existing training mechanism. Hence, the approach is more generalized and can be plugged into any existing FSL method. The pre-trained semantic feature extractor (learned from a large-scale text corpora) we use in our approach provides a strong contextual prior knowledge to assist FSL. The experimental results done in popular FSL datasets show that our approach is generic in nature and provides a strong boost to the existing FSL baselines.

Related papers

• SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification [84.05253637260743]
  We propose a new framework, named Semantic-guided Visual Adapting (SgVA), to extend vision-language pre-trained models. SgVA produces discriminative task-specific visual features by comprehensively using a vision-specific contrastive loss, a cross-modal contrastive loss, and an implicit knowledge distillation. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.
  arXiv  Detail & Related papers  (2022-11-28T14:58:15Z)
• Semantic Cross Attention for Few-shot Learning [9.529264466445236]
  We propose a multi-task learning approach to view semantic features of label text as an auxiliary task. Our proposed model uses word-embedding representations as semantic features to help train the embedding network and a semantic cross-attention module to bridge the semantic features into the typical visual modal.
  arXiv  Detail & Related papers  (2022-10-12T15:24:59Z)
• Brief Introduction to Contrastive Learning Pretext Tasks for Visual Representation [0.0]
  We introduce contrastive learning, a subset of unsupervised learning methods. The purpose of contrastive learning is to embed augmented samples from the same sample near to each other while pushing away those that are not. We offer some strategies from contrastive learning that have recently been published and are focused on pretext tasks for visual representation.
  arXiv  Detail & Related papers  (2022-10-06T18:54:10Z)
• K-LITE: Learning Transferable Visual Models with External Knowledge [242.3887854728843]
  K-LITE (Knowledge-augmented Language-Image Training and Evaluation) is a strategy to leverage external knowledge to build transferable visual systems. In training, it enriches entities in natural language with WordNet and Wiktionary knowledge. In evaluation, the natural language is also augmented with external knowledge and then used to reference learned visual concepts.
  arXiv  Detail & Related papers  (2022-04-20T04:47:01Z)
• Multi-Modal Few-Shot Object Detection with Meta-Learning-Based Cross-Modal Prompting [77.69172089359606]
  We study multi-modal few-shot object detection (FSOD) in this paper, using both few-shot visual examples and class semantic information for detection. Our approach is motivated by the high-level conceptual similarity of (metric-based) meta-learning and prompt-based learning. We comprehensively evaluate the proposed multi-modal FSOD models on multiple few-shot object detection benchmarks, achieving promising results.
  arXiv  Detail & Related papers  (2022-04-16T16:45:06Z)
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [91.56988987393483]
  We present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones.
  arXiv  Detail & Related papers  (2021-12-02T18:59:32Z)
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model [92.2866546058082]
  The visual world naturally exhibits a long-tailed distribution of open classes, which poses great challenges to modern visual systems. Recent advances in contrastive visual-language pretraining shed light on a new pathway for visual recognition. We propose BALLAD to leverage contrastive vision-language models for long-tailed recognition.
  arXiv  Detail & Related papers  (2021-11-29T17:49:24Z)
• Rich Semantics Improve Few-shot Learning [49.11659525563236]
  We show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the few-shot learning performance. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens.
  arXiv  Detail & Related papers  (2021-04-26T16:48:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.