Global Semantic Descriptors for Zero-Shot Action Recognition

• URL: http://arxiv.org/abs/2209.12061v1
• Date: Sat, 24 Sep 2022 18:15:47 GMT
• Title: Global Semantic Descriptors for Zero-Shot Action Recognition
• Abstract summary: This work introduces a new ZSAR method based on the relationships of actions-objects and actions-descriptive sentences. We show how to estimate probabilities over the set of action classes based only on a set of sentences without hard human labeling. Our results are state-of-the-art in the Kinetics-400 dataset and are competitive on UCF-101.
• Score: 2.0206441617430695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of Zero-shot Action Recognition (ZSAR) methods is intrinsically related to the nature of semantic side information used to transfer knowledge, although this aspect has not been primarily investigated in the literature. This work introduces a new ZSAR method based on the relationships of actions-objects and actions-descriptive sentences. We demonstrate that representing all object classes using descriptive sentences generates an accurate object-action affinity estimation when a paraphrase estimation method is used as an embedder. We also show how to estimate probabilities over the set of action classes based only on a set of sentences without hard human labeling. In our method, the probabilities from these two global classifiers (i.e., which use features computed over the entire video) are combined, producing an efficient transfer knowledge model for action classification. Our results are state-of-the-art in the Kinetics-400 dataset and are competitive on UCF-101 under the ZSAR evaluation. Our code is available at https://github.com/valterlej/objsentzsar

Related papers

• CEZSAR: A Contrastive Embedding Method for Zero-Shot Action Recognition [3.971295713554491]
  This paper proposes a novel Zero-Shot Action Recognition(ZSAR) method based on contrastive learning.<n>Two well-known problems remain in ZSAR: the semantic gap and the domain shift.<n>We propose a new model that encodes videos and sentences in a joint embedding space, trained by aligning videos with their natural-language descriptions.
  arXiv  Detail & Related papers  (2026-05-01T23:47:29Z)
• ActivityNarrated: An Open-Ended Narrative Paradigm for Wearable Human Activity Understanding [3.8013366586299093]
  We show how to model open-ended activity narratives by aligning wearable sensor data with natural-language descriptions in an open-vocabulary setting.<n>Results establish open-ended narrative modeling as a practical and effective foundation for real-world wearable HAR.
  arXiv  Detail & Related papers  (2026-04-01T11:31:44Z)
• Description Boosting for Zero-Shot Entity and Relation Classification [5.8959034854546815]
  We show that Zero-Shot Learning (ZSL) methods are sensitive to provided textual descriptions of entities (or relations) We propose a strategy for generating variations of an initial description and an ensemble method capable of boosting the predictions of zero-shot models through description enhancement.
  arXiv  Detail & Related papers  (2024-06-04T12:09:44Z)
• Enhancing Visual Continual Learning with Language-Guided Supervision [76.38481740848434]
  Continual learning aims to empower models to learn new tasks without forgetting previously acquired knowledge. We argue that the scarce semantic information conveyed by the one-hot labels hampers the effective knowledge transfer across tasks. Specifically, we use PLMs to generate semantic targets for each class, which are frozen and serve as supervision signals.
  arXiv  Detail & Related papers  (2024-03-24T12:41:58Z)
• Multi-Semantic Fusion Model for Generalized Zero-Shot Skeleton-Based Action Recognition [32.291333054680855]
  Generalized zero-shot skeleton-based action recognition (GZSSAR) is a new challenging problem in computer vision community. We propose a multi-semantic fusion (MSF) model for improving the performance of GZSSAR.
  arXiv  Detail & Related papers  (2023-09-18T09:00:25Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• Task-Specific Embeddings for Ante-Hoc Explainable Text Classification [6.671252951387647]
  We propose an alternative training objective in which we learn task-specific embeddings of text. Our proposed objective learns embeddings such that all texts that share the same target class label should be close together. We present extensive experiments which show that the benefits of ante-hoc explainability and incremental learning come at no cost in overall classification accuracy.
  arXiv  Detail & Related papers  (2022-11-30T19:56:25Z)
• Cross-modal Representation Learning for Zero-shot Action Recognition [67.57406812235767]
  We present a cross-modal Transformer-based framework, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR) Our model employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner. Experiment results show our model considerably improves upon the state of the arts in ZSAR, reaching encouraging top-1 accuracy on UCF101, HMDB51, and ActivityNet benchmark datasets.
  arXiv  Detail & Related papers  (2022-05-03T17:39:27Z)
• Tell me what you see: A zero-shot action recognition method based on natural language descriptions [3.136605193634262]
  We propose using video captioning methods to extract semantic information from videos. To the best of our knowledge, this is the first work to represent both videos and labels with descriptive sentences. We build a shared semantic space employing BERT-based embedders pre-trained in the paraphrasing task on multiple text datasets.
  arXiv  Detail & Related papers  (2021-12-18T17:44:07Z)
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption [72.35532598131176]
  We propose SCARF, a technique for contrastive learning, where views are formed by corrupting a random subset of features. We show that SCARF complements existing strategies and outperforms alternatives like autoencoders.
  arXiv  Detail & Related papers  (2021-06-29T08:08:33Z)
• GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition [33.23662792742078]
  We propose a two-stage deep neural network for zero-shot action recognition. In the sampling stage, we utilize a generative adversarial networks (GAN) trained by action features and word vectors of seen classes. In the classification stage, we construct a knowledge graph based on the relationship between word vectors of action classes and related objects.
  arXiv  Detail & Related papers  (2021-05-25T09:34:42Z)
• Pairwise Similarity Knowledge Transfer for Weakly Supervised Object Localization [53.99850033746663]
  We study the problem of learning localization model on target classes with weakly supervised image labels. In this work, we argue that learning only an objectness function is a weak form of knowledge transfer. Experiments on the COCO and ILSVRC 2013 detection datasets show that the performance of the localization model improves significantly with the inclusion of pairwise similarity function.
  arXiv  Detail & Related papers  (2020-03-18T17:53:33Z)
• Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification [139.44681304276]
  zero-shot learning aims to classify unseen categories for which no data is available during training. Generative Adrial Networks synthesize unseen class features by leveraging class-specific semantic embeddings. We propose to enforce semantic consistency at all stages of zero-shot learning: training, feature synthesis and classification.
  arXiv  Detail & Related papers  (2020-03-17T17:34:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.