On the Transferability of Visual Features in Generalized Zero-Shot Learning

• URL: http://arxiv.org/abs/2211.12494v1
• Date: Tue, 22 Nov 2022 18:59:09 GMT
• Title: On the Transferability of Visual Features in Generalized Zero-Shot Learning
• Authors: Paola Cascante-Bonilla, Leonid Karlinsky, James Seale Smith, Yanjun Qi, Vicente Ordonez
• Abstract summary: Generalized Zero-Shot Learning (GZSL) aims to train a classifier that can generalize to unseen classes. In this work, we investigate the utility of different GZSL methods when using different feature extractors. We also examine how these models' pre-training objectives, datasets, and architecture design affect their feature representation ability.
• Score: 28.120004119724577
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generalized Zero-Shot Learning (GZSL) aims to train a classifier that can generalize to unseen classes, using a set of attributes as auxiliary information, and the visual features extracted from a pre-trained convolutional neural network. While recent GZSL methods have explored various techniques to leverage the capacity of these features, there has been an extensive growth of representation learning techniques that remain under-explored. In this work, we investigate the utility of different GZSL methods when using different feature extractors, and examine how these models' pre-training objectives, datasets, and architecture design affect their feature representation ability. Our results indicate that 1) methods using generative components for GZSL provide more advantages when using recent feature extractors; 2) feature extractors pre-trained using self-supervised learning objectives and knowledge distillation provide better feature representations, increasing up to 15% performance when used with recent GZSL techniques; 3) specific feature extractors pre-trained with larger datasets do not necessarily boost the performance of GZSL methods. In addition, we investigate how GZSL methods fare against CLIP, a more recent multi-modal pre-trained model with strong zero-shot performance. We found that GZSL tasks still benefit from generative-based GZSL methods along with CLIP's internet-scale pre-training to achieve state-of-the-art performance in fine-grained datasets. We release a modular framework for analyzing representation learning issues in GZSL here: https://github.com/uvavision/TV-GZSL

Related papers

• Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models [53.48409081555687]
  In this work, we explore such large pre-trained models to obtain features, i.e. CLIP for visual features, and CLAP for audio features. We propose a simple yet effective model that only relies on feed-forward neural networks. Our framework achieves state-of-the-art performance on VGGSound-GZSL, UCF-GZSL, and ActivityNet-GZSL.
  arXiv  Detail & Related papers  (2024-04-09T13:39:37Z)
• Disentangled Ontology Embedding for Zero-shot Learning [39.014714187825646]
  Knowledge Graph (KG) and its variant of ontology have been widely used for knowledge representation, and have shown to be quite effective in augmenting Zero-shot Learning (ZSL) Existing ZSL methods that utilize KGs all neglect the complexity of inter-class relationships represented in KGs. In this paper, we focus on for augmenting ZSL, and propose to learn disentangled ontology embeddings guided by semantic properties. We also contribute a new ZSL framework named DOZSL, which contains two new ZSL solutions based on generative models and graph propagation models.
  arXiv  Detail & Related papers  (2022-06-08T08:29:30Z)
• Semantic Feature Extraction for Generalized Zero-shot Learning [23.53412767106488]
  Generalized zero-shot learning (GZSL) is a technique to train a deep learning model to identify unseen classes using the attribute. In this paper, we put forth a new GZSL technique that improves the GZSL classification performance greatly.
  arXiv  Detail & Related papers  (2021-12-29T09:52:30Z)
• Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloud [79.99653758293277]
  We present the first generative approach for both Zero-Shot Learning (ZSL) and Generalized ZSL (GZSL) on 3D data. We show that it reaches or outperforms the state of the art on ModelNet40 classification for both inductive ZSL and inductive GZSL. Our experiments show that our method outperforms strong baselines, which we additionally propose for this task.
  arXiv  Detail & Related papers  (2021-08-13T13:29:27Z)
• Multi-Head Self-Attention via Vision Transformer for Zero-Shot Learning [11.66422653137002]
  We propose an attention-based model in the problem settings of Zero-Shot Learning to learn attributes useful for unseen class recognition. Our method uses an attention mechanism adapted from Vision Transformer to capture and learn discriminative attributes by splitting images into small patches.
  arXiv  Detail & Related papers  (2021-07-30T19:08:44Z)
• FREE: Feature Refinement for Generalized Zero-Shot Learning [86.41074134041394]
  Generalized zero-shot learning (GZSL) has achieved significant progress, with many efforts dedicated to overcoming the problems of visual-semantic domain gap and seen-unseen bias. Most existing methods directly use feature extraction models trained on ImageNet alone, ignoring the cross-dataset bias between ImageNet and GZSL benchmarks. We propose a simple yet effective GZSL method, termed feature refinement for generalized zero-shot learning (FREE) to tackle the above problem.
  arXiv  Detail & Related papers  (2021-07-29T08:11:01Z)
• Attribute-Modulated Generative Meta Learning for Zero-Shot Classification [52.64680991682722]
  We present the Attribute-Modulated generAtive meta-model for Zero-shot learning (AMAZ) Our model consists of an attribute-aware modulation network and an attribute-augmented generative network. Our empirical evaluations show that AMAZ improves state-of-the-art methods by 3.8% and 5.1% in ZSL and generalized ZSL settings, respectively.
  arXiv  Detail & Related papers  (2021-04-22T04:16:43Z)
• Task Aligned Generative Meta-learning for Zero-shot Learning [64.16125851588437]
  We propose a Task-aligned Generative Meta-learning model for Zero-shot learning (TGMZ) TGMZ mitigates the potentially biased training and enables meta-ZSL to accommodate real-world datasets containing diverse distributions. Our comparisons with state-of-the-art algorithms show the improvements of 2.1%, 3.0%, 2.5%, and 7.6% achieved by TGMZ on AWA1, AWA2, CUB, and aPY datasets.
  arXiv  Detail & Related papers  (2021-03-03T05:18:36Z)
• OntoZSL: Ontology-enhanced Zero-shot Learning [19.87808305218359]
  Key to implementing Zero-shot Learning (ZSL) is to leverage the prior knowledge of classes which builds the semantic relationship between classes. In this paper, we explore richer and more competitive prior knowledge to model the inter-class relationship for ZSL. To address the data imbalance between seen classes and unseen classes, we developed a generative ZSL framework with Generative Adversarial Networks (GANs)
  arXiv  Detail & Related papers  (2021-02-15T04:39:58Z)
• End-to-end Generative Zero-shot Learning via Few-shot Learning [76.9964261884635]
  State-of-the-art approaches to Zero-Shot Learning (ZSL) train generative nets to synthesize examples conditioned on the provided metadata. We introduce an end-to-end generative ZSL framework that uses such an approach as a backbone and feeds its synthesized output to a Few-Shot Learning algorithm.
  arXiv  Detail & Related papers  (2021-02-08T17:35:37Z)
• A Biologically Inspired Feature Enhancement Framework for Zero-Shot Learning [18.348568695197553]
  This paper proposes a biologically inspired feature enhancement framework for Zero-Shot Learning (ZSL) algorithms. Specifically, we design a dual-channel learning framework that uses auxiliary data sets to enhance the feature extractor of the ZSL model. Our proposed method can effectively improve the ability of the ZSL model and achieve state-of-the-art results on three benchmark ZSL tasks.
  arXiv  Detail & Related papers  (2020-05-13T13:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.