Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition

• URL: http://arxiv.org/abs/2405.18012v1
• Date: Tue, 28 May 2024 09:53:47 GMT
• Title: Flow-Assisted Motion Learning Network for Weakly-Supervised Group Activity Recognition
• Authors: Muhammad Adi Nugroho, Sangmin Woo, Sumin Lee, Jinyoung Park, Yooseung Wang, Donguk Kim, Changick Kim,
• Abstract summary: Weakly-Supervised Group Activity Recognition (WSGAR) aims to understand the activity performed together by a group of individuals with the video-level label and without actor-level labels. We propose Flow-Assisted Motion Learning Network (Flaming-Net) for WSGAR, which consists of the motion-aware encoder to extract actor features. We demonstrate that Flaming-Net new state-of-the-art WSGAR results on two benchmarks, including a 2.8%p higher MPCA score on the NBA dataset.
• Score: 21.482797499764093
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weakly-Supervised Group Activity Recognition (WSGAR) aims to understand the activity performed together by a group of individuals with the video-level label and without actor-level labels. We propose Flow-Assisted Motion Learning Network (Flaming-Net) for WSGAR, which consists of the motion-aware actor encoder to extract actor features and the two-pathways relation module to infer the interaction among actors and their activity. Flaming-Net leverages an additional optical flow modality in the training stage to enhance its motion awareness when finding locally active actors. The first pathway of the relation module, the actor-centric path, initially captures the temporal dynamics of individual actors and then constructs inter-actor relationships. In parallel, the group-centric path starts by building spatial connections between actors within the same timeframe and then captures simultaneous spatio-temporal dynamics among them. We demonstrate that Flaming-Net achieves new state-of-the-art WSGAR results on two benchmarks, including a 2.8%p higher MPCA score on the NBA dataset. Importantly, we use the optical flow modality only for training and not for inference.

Related papers

• PaCMO: Partner Dependent Human Motion Generation in Dyadic Human Activity using Neural Operators [20.45590914720127]
  We propose a neural operator-based generative model which learns the distribution of human motion conditioned by the partner's motion in function spaces. Our model can handle long unlabeled action sequences at arbitrary time resolution. We test PaCMO on NTU RGB+D and DuetDance datasets and our model produces realistic results.
  arXiv  Detail & Related papers  (2022-11-25T22:20:11Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)
• Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition [103.62363658053557]
  We propose a Dual-path Actor Interaction (DualAI) framework, which flexibly arranges spatial and temporal transformers. We also introduce a novel Multi-scale Actor Contrastive Loss (MAC-Loss) between two interactive paths of Dual-AI. Our Dual-AI can boost group activity recognition by fusing distinct discriminative features of different actors.
  arXiv  Detail & Related papers  (2022-04-05T12:17:40Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• Learning Comprehensive Motion Representation for Action Recognition [124.65403098534266]
  2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing inter-frame connections while still suffering the limited temporal receptive field or high latency. We propose a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. We also propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps.
  arXiv  Detail & Related papers  (2021-03-23T03:06:26Z)
• Learning Asynchronous and Sparse Human-Object Interaction in Videos [56.73059840294019]
  Asynchronous-Sparse Interaction Graph Networks (ASSIGN) is able to automatically detect the structure of interaction events associated with entities in a video scene. ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos.
  arXiv  Detail & Related papers  (2021-03-03T23:43:55Z)
• Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video [64.44583693846751]
  We study the semi-supervised instrument segmentation from robotic surgical videos with sparse annotations. By exploiting generated data pairs, our framework can recover and even enhance temporal consistency of training sequences. Results show that our method outperforms the state-of-the-art semisupervised methods by a large margin.
  arXiv  Detail & Related papers  (2020-07-06T02:39:32Z)
• Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization [47.61419011906561]
  ACAR-Net builds upon a novel High-order Relation Reasoning Operator to enable indirect reasoning fortemporal action localization. Our method ranks first in the AVA-Kineticsaction localization task of ActivityNet Challenge 2020.
  arXiv  Detail & Related papers  (2020-06-14T18:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.