One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Features

• URL: http://arxiv.org/abs/2404.19542v1
• Date: Tue, 30 Apr 2024 13:14:28 GMT
• Title: One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Features
• Authors: Trung Thanh Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide,
• Abstract summary: Open-vocabulary Temporal Action Detection (Open-vocab TAD) is an advanced video analysis approach. The proposed method achieves superior results compared to the other methods in both Open-vocab and Closed-vocab settings.
• Score: 2.8266810371534152
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary Temporal Action Detection (Open-vocab TAD) is an advanced video analysis approach that expands Closed-vocabulary Temporal Action Detection (Closed-vocab TAD) capabilities. Closed-vocab TAD is typically confined to localizing and classifying actions based on a predefined set of categories. In contrast, Open-vocab TAD goes further and is not limited to these predefined categories. This is particularly useful in real-world scenarios where the variety of actions in videos can be vast and not always predictable. The prevalent methods in Open-vocab TAD typically employ a 2-stage approach, which involves generating action proposals and then identifying those actions. However, errors made during the first stage can adversely affect the subsequent action identification accuracy. Additionally, existing studies face challenges in handling actions of different durations owing to the use of fixed temporal processing methods. Therefore, we propose a 1-stage approach consisting of two primary modules: Multi-scale Video Analysis (MVA) and Video-Text Alignment (VTA). The MVA module captures actions at varying temporal resolutions, overcoming the challenge of detecting actions with diverse durations. The VTA module leverages the synergy between visual and textual modalities to precisely align video segments with corresponding action labels, a critical step for accurate action identification in Open-vocab scenarios. Evaluations on widely recognized datasets THUMOS14 and ActivityNet-1.3, showed that the proposed method achieved superior results compared to the other methods in both Open-vocab and Closed-vocab settings. This serves as a strong demonstration of the effectiveness of the proposed method in the TAD task.

Related papers

• Technical Report for ActivityNet Challenge 2022 -- Temporal Action Localization [20.268572246761895]
  We propose to locate the temporal boundaries of each action and predict action class in untrimmed videos. Faster-TAD simplifies the pipeline of TAD and gets remarkable performance.
  arXiv  Detail & Related papers  (2024-10-31T14:16:56Z)
• Harnessing Temporal Causality for Advanced Temporal Action Detection [53.654457142657236]
  We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on benchmarks. We ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, and 1st in the Moment Queries track at the Ego4D Challenge 2024.
  arXiv  Detail & Related papers  (2024-07-25T06:03:02Z)
• ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos [35.371453530275666]
  ActionSwitch is the first class-agnostic On-TAL framework capable of detecting overlapping actions. By obviating the reliance on class information, ActionSwitch provides wider applicability to various situations.
  arXiv  Detail & Related papers  (2024-07-17T20:07:05Z)
• Weakly-Supervised Temporal Action Localization with Bidirectional Semantic Consistency Constraint [83.36913240873236]
  Weakly Supervised Temporal Action localization (WTAL) aims to classify and localize temporal boundaries of actions for the video. We propose a simple yet efficient method, named bidirectional semantic consistency constraint (Bi- SCC) to discriminate the positive actions from co-scene actions. Experimental results show that our approach outperforms the state-of-the-art methods on THUMOS14 and ActivityNet.
  arXiv  Detail & Related papers  (2023-04-25T07:20:33Z)
• Multi-modal Prompting for Low-Shot Temporal Action Localization [95.19505874963751]
  We consider the problem of temporal action localization under low-shot (zero-shot & few-shot) scenario. We adopt a Transformer-based two-stage action localization architecture with class-agnostic action proposal, followed by open-vocabulary classification.
  arXiv  Detail & Related papers  (2023-03-21T10:40:13Z)
• SegTAD: Precise Temporal Action Detection via Semantic Segmentation [65.01826091117746]
  We formulate the task of temporal action detection in a novel perspective of semantic segmentation. Owing to the 1-dimensional property of TAD, we are able to convert the coarse-grained detection annotations to fine-grained semantic segmentation annotations for free. We propose an end-to-end framework SegTAD composed of a 1D semantic segmentation network (1D-SSN) and a proposal detection network (PDN)
  arXiv  Detail & Related papers  (2022-03-03T06:52:13Z)
• End-to-end Temporal Action Detection with Transformer [86.80289146697788]
  Temporal action detection (TAD) aims to determine the semantic label and the boundaries of every action instance in an untrimmed video. Here, we construct an end-to-end framework for TAD upon Transformer, termed textitTadTR. Our method achieves state-of-the-art performance on HACS Segments and THUMOS14 and competitive performance on ActivityNet-1.3.
  arXiv  Detail & Related papers  (2021-06-18T17:58:34Z)
• Weakly Supervised Temporal Action Localization Through Learning Explicit Subspaces for Action and Context [151.23835595907596]
  Methods learn to localize temporal starts and ends of action instances in a video under only video-level supervision. We introduce a framework that learns two feature subspaces respectively for actions and their context. The proposed approach outperforms state-of-the-art WS-TAL methods on three benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T08:26:53Z)
• Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting [22.86745487695168]
  We propose a baseline based on multi-instance and multi-label learning. We propose a novel approach that uses sets of actions as representation instead of modeling individual action classes. We evaluate the proposed approach on the challenging dataset where the proposed approach outperforms the MIML baseline and is competitive to fully supervised approaches.
  arXiv  Detail & Related papers  (2021-01-21T11:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.