Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning

• URL: http://arxiv.org/abs/2405.03109v1
• Date: Mon, 6 May 2024 02:02:57 GMT
• Title: Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning
• Authors: Weihao Jiang, Chang Liu, Kun He,
• Abstract summary: Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
• Score: 12.5354658533836
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. Such ability stems from their capacity to identify common features shared between new and previously seen images while disregarding distractions such as background variations. However, for artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. In this paper, we propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches and encoding them using the pre-trained Vision Transformer (ViT) architecture. Specifically, we swap the class (CLS) token and patch tokens between the support and query sets to have the mutual attention, which enables each set to focus on the most useful information. This facilitates the strengthening of intra-class representations and promotes closer proximity between instances of the same class. For implementation, we adopt the ViT-based network architecture and utilize pre-trained model parameters obtained through self-supervision. By leveraging Masked Image Modeling as a self-supervised training task for pre-training, the pre-trained model yields semantically meaningful representations while successfully avoiding supervision collapse. We then employ a meta-learning method to fine-tune the last several layers and CLS token modules. Our strategy significantly reduces the num- ber of parameters that require fine-tuning while effectively uti- lizing the capability of pre-trained model. Extensive experiments show that our framework is simple, effective and computationally efficient, achieving superior performance as compared to the state-of-the-art baselines on five popular few-shot classification benchmarks under the 5-shot and 1-shot scenarios

Related papers

• Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach [87.8330887605381]
  We show how to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. We synthesize a task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. Our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.
  arXiv  Detail & Related papers  (2024-07-09T15:45:04Z)
• Class Incremental Learning with Pre-trained Vision-Language Models [59.15538370859431]
  We propose an approach to exploiting pre-trained vision-language models (e.g. CLIP) that enables further adaptation. Experiments on several conventional benchmarks consistently show a significant margin of improvement over the current state-of-the-art.
  arXiv  Detail & Related papers  (2023-10-31T10:45:03Z)
• Heuristic Vision Pre-Training with Self-Supervised and Supervised Multi-Task Learning [0.0]
  We propose a novel pre-training framework by adopting both self-supervised and supervised visual pre-text tasks in a multi-task manner. Results show that our pre-trained models can deliver results on par with or better than state-of-the-art (SOTA) results on multiple visual tasks.
  arXiv  Detail & Related papers  (2023-10-11T14:06:04Z)
• MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments [72.6405488990753]
  Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks. We propose a single-stage and standalone method, MOCA, which unifies both desired properties. We achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols.
  arXiv  Detail & Related papers  (2023-07-18T15:46:20Z)
• Multi-Modal Few-Shot Temporal Action Detection [157.96194484236483]
  Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection to new classes. We introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD.
  arXiv  Detail & Related papers  (2022-11-27T18:13:05Z)
• Masked Unsupervised Self-training for Zero-shot Image Classification [98.23094305347709]
  Masked Unsupervised Self-Training (MUST) is a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification.
  arXiv  Detail & Related papers  (2022-06-07T02:03:06Z)
• Few Shot Activity Recognition Using Variational Inference [9.371378627575883]
  We propose a novel variational inference based architectural framework (HF-AR) for few shot activity recognition. Our framework leverages volume-preserving Householder Flow to learn a flexible posterior distribution of the novel classes. This results in better performance as compared to state-of-the-art few shot approaches for human activity recognition.
  arXiv  Detail & Related papers  (2021-08-20T03:57:58Z)
• Multimodal Contrastive Training for Visual Representation Learning [45.94662252627284]
  We develop an approach to learning visual representations that embraces multimodal data. Our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously. By including multimodal training in a unified framework, our method can learn more powerful and generic visual features.
  arXiv  Detail & Related papers  (2021-04-26T19:23:36Z)
• Region Comparison Network for Interpretable Few-shot Image Classification [97.97902360117368]
  Few-shot image classification has been proposed to effectively use only a limited number of labeled examples to train models for new classes. We propose a metric learning based method named Region Comparison Network (RCN), which is able to reveal how few-shot learning works. We also present a new way to generalize the interpretability from the level of tasks to categories.
  arXiv  Detail & Related papers  (2020-09-08T07:29:05Z)
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases [34.02639091680309]
  Recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet.
  arXiv  Detail & Related papers  (2020-07-28T00:11:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.