Knowledge Augmented Relation Inference for Group Activity Recognition

• URL: http://arxiv.org/abs/2302.14350v2
• Date: Wed, 1 Mar 2023 08:12:08 GMT
• Title: Knowledge Augmented Relation Inference for Group Activity Recognition
• Authors: Xianglong Lang, Zhuming Wang, Zun Li, Meng Tian, Ge Shi, Lifang Wu and Liang Wang
• Abstract summary: We propose to exploit knowledge concretization for the group activity recognition. We develop a novel Knowledge Augmented Relation Inference framework that can effectively use the concretized knowledge to improve the individual representations.
• Score: 14.240856072486666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most existing group activity recognition methods construct spatial-temporal relations merely based on visual representation. Some methods introduce extra knowledge, such as action labels, to build semantic relations and use them to refine the visual presentation. However, the knowledge they explored just stay at the semantic-level, which is insufficient for pursing notable accuracy. In this paper, we propose to exploit knowledge concretization for the group activity recognition, and develop a novel Knowledge Augmented Relation Inference framework that can effectively use the concretized knowledge to improve the individual representations. Specifically, the framework consists of a Visual Representation Module to extract individual appearance features, a Knowledge Augmented Semantic Relation Module explore semantic representations of individual actions, and a Knowledge-Semantic-Visual Interaction Module aims to integrate visual and semantic information by the knowledge. Benefiting from these modules, the proposed framework can utilize knowledge to enhance the relation inference process and the individual representations, thus improving the performance of group activity recognition. Experimental results on two public datasets show that the proposed framework achieves competitive performance compared with state-of-the-art methods.

Related papers

• LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching [25.883546163390957]
  We endow CLIP with fine-grained action-level understanding by incorporating action-related external knowledge generated by large language models (LLMs)<n>We propose an adaptive interaction module to aggregate attentive visual features conditioned on action-aware prompted knowledge for establishing discriminative and action-aware visual representations.
  arXiv  Detail & Related papers  (2025-06-30T03:49:08Z)
• Including Semantic Information via Word Embeddings for Skeleton-based Action Recognition [7.441242294426765]
  We introduce a novel approach to skeleton-based action recognition that enriches input representations by leveraging word embeddings to encode semantic information.<n>Our method replaces one-hot encodings with semantic volumes, enabling the model to capture meaningful relationships between joints and objects.
  arXiv  Detail & Related papers  (2025-06-23T14:57:06Z)
• Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning [58.73625654718187]
  Generalized zero-shot learning aims to recognize both seen and unseen classes with the help of semantic information that is shared among different classes. Existing approaches fine-tune the visual backbone by seen-class data to obtain semantic-related visual features. This paper proposes a novel visual and semantic prompt collaboration framework, which utilizes prompt tuning techniques for efficient feature adaptation.
  arXiv  Detail & Related papers  (2025-03-29T10:17:57Z)
• A Concept-Centric Approach to Multi-Modality Learning [3.828996378105142]
  We introduce a new multi-modality learning framework to create a more efficient AI system. Our framework achieves on par with benchmark models while demonstrating more efficient learning curves.
  arXiv  Detail & Related papers  (2024-12-18T13:40:21Z)
• Augmented Commonsense Knowledge for Remote Object Grounding [67.30864498454805]
  We propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as atemporal knowledge graph for improving agent navigation. ACK consists of knowledge graph-aware cross-modal and concept aggregation modules to enhance visual representation and visual-textual data alignment. We add a new pipeline for the commonsense-based decision-making process which leads to more accurate local action prediction.
  arXiv  Detail & Related papers  (2024-06-03T12:12:33Z)
• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Self-Supervised Visual Representation Learning with Semantic Grouping [50.14703605659837]
  We tackle the problem of learning visual representations from unlabeled scene-centric data. We propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning.
  arXiv  Detail & Related papers  (2022-05-30T17:50:59Z)
• An Empirical Investigation of Representation Learning for Imitation [76.48784376425911]
  Recent work in vision, reinforcement learning, and NLP has shown that auxiliary representation learning objectives can reduce the need for large amounts of expensive, task-specific data. We propose a modular framework for constructing representation learning algorithms, then use our framework to evaluate the utility of representation learning for imitation.
  arXiv  Detail & Related papers  (2022-05-16T11:23:42Z)
• Cross-modal Representation Learning for Zero-shot Action Recognition [67.57406812235767]
  We present a cross-modal Transformer-based framework, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR) Our model employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner. Experiment results show our model considerably improves upon the state of the arts in ZSAR, reaching encouraging top-1 accuracy on UCF101, HMDB51, and ActivityNet benchmark datasets.
  arXiv  Detail & Related papers  (2022-05-03T17:39:27Z)
• Semantic TrueLearn: Using Semantic Knowledge Graphs in Recommendation Systems [22.387120578306277]
  This work aims to advance towards building a state-aware educational recommendation system that incorporates semantic relatedness. We introduce a novel learner model that exploits this semantic relatedness between knowledge components in learning resources using the Wikipedia link graph. Our experiments with a large dataset demonstrate that this new semantic version of TrueLearn algorithm achieves statistically significant improvements in terms of predictive performance.
  arXiv  Detail & Related papers  (2021-12-08T16:23:27Z)
• Learning semantic Image attributes using Image recognition and knowledge graph embeddings [0.3222802562733786]
  We propose a shared learning approach to learn semantic attributes of images by combining a knowledge graph embedding model with the recognized attributes of images. The proposed approach is a step towards bridging the gap between frameworks which learn from large amounts of data and frameworks which use a limited set of predicates to infer new knowledge.
  arXiv  Detail & Related papers  (2020-09-12T15:18:48Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.