Self-supervised Multi-actor Social Activity Understanding in Streaming Videos

• URL: http://arxiv.org/abs/2406.14472v1
• Date: Thu, 20 Jun 2024 16:33:54 GMT
• Title: Self-supervised Multi-actor Social Activity Understanding in Streaming Videos
• Authors: Shubham Trehan, Sathyanarayanan N. Aakur,
• Abstract summary: Social Activity Recognition (SAR) is a critical component in real-world tasks like surveillance and assistive robotics. Previous SAR research has relied heavily on densely annotated data, but privacy concerns limit their applicability in real-world settings. We propose a self-supervised approach based on multi-actor predictive learning for SAR in streaming videos.
• Score: 6.4149117677272525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work addresses the problem of Social Activity Recognition (SAR), a critical component in real-world tasks like surveillance and assistive robotics. Unlike traditional event understanding approaches, SAR necessitates modeling individual actors' appearance and motions and contextualizing them within their social interactions. Traditional action localization methods fall short due to their single-actor, single-action assumption. Previous SAR research has relied heavily on densely annotated data, but privacy concerns limit their applicability in real-world settings. In this work, we propose a self-supervised approach based on multi-actor predictive learning for SAR in streaming videos. Using a visual-semantic graph structure, we model social interactions, enabling relational reasoning for robust performance with minimal labeled data. The proposed framework achieves competitive performance on standard group activity recognition benchmarks. Evaluation on three publicly available action localization benchmarks demonstrates its generalizability to arbitrary action localization.

Related papers

• UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark [20.15425745473231]
  Localizing unusual activities, such as human errors or surveillance incidents, in videos holds practical significance. To explore foundation models' capability in localizing unusual activity, we introduce UAL-Bench. UAL-Bench features three video datasets: UAG-OOPS, UAG- SSBD, UAG-FunQA, and an instruction-tune dataset: OOPS-UAG-Instruct. Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs.
  arXiv  Detail & Related papers  (2024-10-02T02:33:09Z)
• Learning Where to Look: Self-supervised Viewpoint Selection for Active Localization using Geometrical Information [68.10033984296247]
  This paper explores the domain of active localization, emphasizing the importance of viewpoint selection to enhance localization accuracy. Our contributions involve using a data-driven approach with a simple architecture designed for real-time operation, a self-supervised data training method, and the capability to consistently integrate our map into a planning framework tailored for real-world robotics applications.
  arXiv  Detail & Related papers  (2024-07-22T12:32:09Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• REACT: Recognize Every Action Everywhere All At Once [8.10024991952397]
  Group Activity Decoder (GAR) is a fundamental problem in computer vision, with diverse applications in sports analysis, surveillance, and social scene understanding. We present REACT, an architecture inspired by the transformer encoder-decoder model. Our method outperforms state-of-the-art GAR approaches in extensive experiments, demonstrating superior accuracy in recognizing and understanding group activities.
  arXiv  Detail & Related papers  (2023-11-27T20:48:54Z)
• Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition [111.87412719773889]
  We propose a joint learning framework for "interacted object localization" and "human action recognition" based on skeleton data. Our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition.
  arXiv  Detail & Related papers  (2021-10-28T10:09:34Z)
• Elaborative Rehearsal for Zero-shot Action Recognition [36.84404523161848]
  ZSAR aims to recognize target (unseen) actions without training examples. It remains challenging to semantically represent action classes and transfer knowledge from seen data. We propose an ER-enhanced ZSAR model inspired by an effective human memory technique Elaborative Rehearsal.
  arXiv  Detail & Related papers  (2021-08-05T20:02:46Z)
• JRDB-Act: A Large-scale Multi-modal Dataset for Spatio-temporal Action, Social Group and Activity Detection [54.696819174421584]
  We introduce JRDB-Act, a multi-modal dataset that reflects a real distribution of human daily life actions in a university campus environment. JRDB-Act has been densely annotated with atomic actions, comprises over 2.8M action labels. JRDB-Act comes with social group identification annotations conducive to the task of grouping individuals based on their interactions in the scene.
  arXiv  Detail & Related papers  (2021-06-16T14:43:46Z)
• Learning Actor-centered Representations for Action Localization in Streaming Videos using Predictive Learning [18.757368441841123]
  Event perception tasks such as recognizing and localizing actions in streaming videos are essential for tackling visual understanding tasks. We tackle the problem of learning textitactor-centered representations through the notion of continual hierarchical predictive learning. Inspired by cognitive theories of event perception, we propose a novel, self-supervised framework.
  arXiv  Detail & Related papers  (2021-04-29T06:06:58Z)
• Learning Asynchronous and Sparse Human-Object Interaction in Videos [56.73059840294019]
  Asynchronous-Sparse Interaction Graph Networks (ASSIGN) is able to automatically detect the structure of interaction events associated with entities in a video scene. ASSIGN is tested on human-object interaction recognition and shows superior performance in segmenting and labeling of human sub-activities and object affordances from raw videos.
  arXiv  Detail & Related papers  (2021-03-03T23:43:55Z)
• Self-supervised Human Detection and Segmentation via Multi-view Consensus [116.92405645348185]
  We propose a multi-camera framework in which geometric constraints are embedded in the form of multi-view consistency during training. We show that our approach outperforms state-of-the-art self-supervised person detection and segmentation techniques on images that visually depart from those of standard benchmarks.
  arXiv  Detail & Related papers  (2020-12-09T15:47:21Z)
• Attention-Oriented Action Recognition for Real-Time Human-Robot Interaction [11.285529781751984]
  We propose an attention-oriented multi-level network framework to meet the need for real-time interaction. Specifically, a Pre-Attention network is employed to roughly focus on the interactor in the scene at low resolution. The other compact CNN receives the extracted skeleton sequence as input for action recognition.
  arXiv  Detail & Related papers  (2020-07-02T12:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.