GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition

• URL: http://arxiv.org/abs/2105.11789v1
• Date: Tue, 25 May 2021 09:34:42 GMT
• Title: GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition
• Authors: Bin Sun, Dehui Kong, Shaofan Wang, Jinghua Li, Baocai Yin, Xiaonan Luo
• Abstract summary: We propose a two-stage deep neural network for zero-shot action recognition. In the sampling stage, we utilize a generative adversarial networks (GAN) trained by action features and word vectors of seen classes. In the classification stage, we construct a knowledge graph based on the relationship between word vectors of action classes and related objects.
• Score: 33.23662792742078
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot action recognition can recognize samples of unseen classes that are unavailable in training by exploring common latent semantic representation in samples. However, most methods neglected the connotative relation and extensional relation between the action classes, which leads to the poor generalization ability of the zero-shot learning. Furthermore, the learned classifier incline to predict the samples of seen class, which leads to poor classification performance. To solve the above problems, we propose a two-stage deep neural network for zero-shot action recognition, which consists of a feature generation sub-network serving as the sampling stage and a graph attention sub-network serving as the classification stage. In the sampling stage, we utilize a generative adversarial networks (GAN) trained by action features and word vectors of seen classes to synthesize the action features of unseen classes, which can balance the training sample data of seen classes and unseen classes. In the classification stage, we construct a knowledge graph (KG) based on the relationship between word vectors of action classes and related objects, and propose a graph convolution network (GCN) based on attention mechanism, which dynamically updates the relationship between action classes and objects, and enhances the generalization ability of zero-shot learning. In both stages, we all use word vectors as bridges for feature generation and classifier generalization from seen classes to unseen classes. We compare our method with state-of-the-art methods on UCF101 and HMDB51 datasets. Experimental results show that our proposed method improves the classification performance of the trained classifier and achieves higher accuracy.

Related papers

• Investigating Self-Supervised Methods for Label-Efficient Learning [27.029542823306866]
  We study different self supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling for their low-shot capabilities. We introduce a framework involving both mask image modelling and clustering as pretext tasks, which performs better across all low-shot downstream tasks. When testing the model on full scale datasets, we show performance gains in multi-class classification, multi-label classification and semantic segmentation.
  arXiv  Detail & Related papers  (2024-06-25T10:56:03Z)
• Fine-Grained Visual Classification using Self Assessment Classifier [12.596520707449027]
  Extracting discriminative features plays a crucial role in the fine-grained visual classification task. In this paper, we introduce a Self Assessment, which simultaneously leverages the representation of the image and top-k prediction classes. We show that our method achieves new state-of-the-art results on CUB200-2011, Stanford Dog, and FGVC Aircraft datasets.
  arXiv  Detail & Related papers  (2022-05-21T07:41:27Z)
• Semantic Representation and Dependency Learning for Multi-Label Image Recognition [76.52120002993728]
  We propose a novel and effective semantic representation and dependency learning (SRDL) framework to learn category-specific semantic representation for each category. Specifically, we design a category-specific attentional regions (CAR) module to generate channel/spatial-wise attention matrices to guide model. We also design an object erasing (OE) module to implicitly learn semantic dependency among categories by erasing semantic-aware regions.
  arXiv  Detail & Related papers  (2022-04-08T00:55:15Z)
• The Overlooked Classifier in Human-Object Interaction Recognition [82.20671129356037]
  We encode the semantic correlation among classes into the classification head by initializing the weights with language embeddings of HOIs. We propose a new loss named LSE-Sign to enhance multi-label learning on a long-tailed dataset. Our simple yet effective method enables detection-free HOI classification, outperforming the state-of-the-arts that require object detection and human pose by a clear margin.
  arXiv  Detail & Related papers  (2022-03-10T23:35:00Z)
• Learning Debiased and Disentangled Representations for Semantic Segmentation [52.35766945827972]
  We propose a model-agnostic and training scheme for semantic segmentation. By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes. Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-10-31T16:15:09Z)
• CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action Recognition [52.66360172784038]
  We propose a clustering-based model, which considers all training samples at once, instead of optimizing for each instance individually. We call the proposed method CLASTER and observe that it consistently improves over the state-of-the-art in all standard datasets.
  arXiv  Detail & Related papers  (2021-01-18T12:46:24Z)
• Adaptive Prototypical Networks with Label Words and Joint Representation Learning for Few-Shot Relation Classification [17.237331828747006]
  This work focuses on few-shot relation classification (FSRC) We propose an adaptive mixture mechanism to add label words to the representation of the class prototype. Experiments have been conducted on FewRel under different few-shot (FS) settings.
  arXiv  Detail & Related papers  (2021-01-10T11:25:42Z)
• Learning and Evaluating Representations for Deep One-class Classification [59.095144932794646]
  We present a two-stage framework for deep one-class classification. We first learn self-supervised representations from one-class data, and then build one-class classifiers on learned representations. In experiments, we demonstrate state-of-the-art performance on visual domain one-class classification benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T23:33:41Z)
• Attribute Propagation Network for Graph Zero-shot Learning [57.68486382473194]
  We introduce the attribute propagation network (APNet), which is composed of 1) a graph propagation model generating attribute vector for each class and 2) a parameterized nearest neighbor (NN) classifier. APNet achieves either compelling performance or new state-of-the-art results in experiments with two zero-shot learning settings and five benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-24T16:53:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.