DOAD: Decoupled One Stage Action Detection Network

• URL: http://arxiv.org/abs/2304.00254v2
• Date: Tue, 4 Apr 2023 07:08:41 GMT
• Title: DOAD: Decoupled One Stage Action Detection Network
• Authors: Shuning Chang, Pichao Wang, Fan Wang, Jiashi Feng, Mike Zheng Show
• Abstract summary: Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. We present a decoupled one-stage network dubbed DOAD, to improve the efficiency for-temporal action detection.
• Score: 77.14883592642782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. However, such two-stage methods are generally with low efficiency. We observe that directly unifying detection and action recognition normally suffers from (i) inferior learning due to different desired properties of context representation for detection and action recognition; (ii) optimization difficulty with insufficient training data. In this work, we present a decoupled one-stage network dubbed DOAD, to mitigate above issues and improve the efficiency for spatio-temporal action detection. To achieve it, we decouple detection and action recognition into two branches. Specifically, one branch focuses on detection representation for actor detection, and the other one for action recognition. For the action branch, we design a transformer-based module (TransPC) to model pairwise relationships between people and context. Different from commonly used vector-based dot product in self-attention, it is built upon a novel matrix-based key and value for Hadamard attention to model person-context information. It not only exploits relationships between person pairs but also takes into account context and relative position information. The results on AVA and UCF101-24 datasets show that our method is competitive with two-stage state-of-the-art methods with significant efficiency improvement.

Related papers

• JARViS: Detecting Actions in Video Using Unified Actor-Scene Context Relation Modeling [8.463489896549161]
  Two-stage Video localization (VAD) is a formidable task that involves the localization and classification of actions within the spatial and temporal dimensions of a video clip. We propose a two-stage VAD framework called Joint Actor-scene context Relation modeling (JARViS) JARViS consolidates cross-modal action semantics distributed globally across spatial and temporal dimensions using Transformer attention.
  arXiv  Detail & Related papers  (2024-08-07T08:08:08Z)
• The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based tasks [4.971065912401385]
  We propose Dual-VCLIP, a unified approach for zero-shot multi-label action recognition. Dual-VCLIP enhances VCLIP, a zero-shot action recognition method, with the DualCoOp method for multi-label image classification. We validate our method on the Charades dataset that includes a majority of object-based actions.
  arXiv  Detail & Related papers  (2024-05-14T15:28:48Z)
• Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy [12.257725479880458]
  Action recognition has become one of the popular research topics in computer vision. We propose a multi-view attention consistency method that computes the similarity between two attentions from two different views of the action videos. Our approach applies the idea of Neural Radiance Field to implicitly render the features from novel views when training on single-view datasets.
  arXiv  Detail & Related papers  (2024-05-02T14:43:21Z)
• Knowledge Guided Bidirectional Attention Network for Human-Object Interaction Detection [3.0915392100355192]
  We argue that the independent use of the bottom-up parsing strategy in HOI is counter-intuitive and could lead to the diffusion of attention. We introduce a novel knowledge-guided top-down attention into HOI, and propose to model the relation parsing as a "look and search" process. We implement the process via unifying the bottom-up and top-down attention in a single encoder-decoder based model.
  arXiv  Detail & Related papers  (2022-07-16T16:42:49Z)
• Few-Shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning [51.03781020616402]
  Fine-grained action recognition is attracting increasing attention due to the emerging demand of specific action understanding in real-world applications. We propose a few-shot fine-grained action recognition problem, aiming to recognize novel fine-grained actions with only few samples given for each class. Although progress has been made in coarse-grained actions, existing few-shot recognition methods encounter two issues handling fine-grained actions.
  arXiv  Detail & Related papers  (2021-08-15T02:21:01Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• Unsupervised Pretraining for Object Detection by Patch Reidentification [72.75287435882798]
  Unsupervised representation learning achieves promising performances in pre-training representations for object detectors. This work proposes a simple yet effective representation learning method for object detection, named patch re-identification (Re-ID) Our method significantly outperforms its counterparts on COCO in all settings, such as different training iterations and data percentages.
  arXiv  Detail & Related papers  (2021-03-08T15:13:59Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)
• Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition [86.31412529187243]
  Few-shot video recognition aims at learning new actions with only very few labeled samples. We propose a depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net.
  arXiv  Detail & Related papers  (2020-10-20T03:06:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.