Open-Vocabulary Spatio-Temporal Action Detection

• URL: http://arxiv.org/abs/2405.10832v1
• Date: Fri, 17 May 2024 14:52:47 GMT
• Title: Open-Vocabulary Spatio-Temporal Action Detection
• Authors: Tao Wu, Shuqiu Ge, Jie Qin, Gangshan Wu, Limin Wang,
• Abstract summary: Open-vocabulary-temporal action detection (OV-STAD) is an important fine-grained video understanding task. OV-STAD requires training a model on a limited set of base classes with box and label supervision. To better adapt the holistic VLM for the fine-grained action detection task, we carefully fine-tune it on the localized video region-text pairs.
• Score: 59.91046192096296
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Spatio-temporal action detection (STAD) is an important fine-grained video understanding task. Current methods require box and label supervision for all action classes in advance. However, in real-world applications, it is very likely to come across new action classes not seen in training because the action category space is large and hard to enumerate. Also, the cost of data annotation and model training for new classes is extremely high for traditional methods, as we need to perform detailed box annotations and re-train the whole network from scratch. In this paper, we propose a new challenging setting by performing open-vocabulary STAD to better mimic the situation of action detection in an open world. Open-vocabulary spatio-temporal action detection (OV-STAD) requires training a model on a limited set of base classes with box and label supervision, which is expected to yield good generalization performance on novel action classes. For OV-STAD, we build two benchmarks based on the existing STAD datasets and propose a simple but effective method based on pretrained video-language models (VLM). To better adapt the holistic VLM for the fine-grained action detection task, we carefully fine-tune it on the localized video region-text pairs. This customized fine-tuning endows the VLM with better motion understanding, thus contributing to a more accurate alignment between video regions and texts. Local region feature and global video feature fusion before alignment is adopted to further improve the action detection performance by providing global context. Our method achieves a promising performance on novel classes.

Related papers

• ActPrompt: In-Domain Feature Adaptation via Action Cues for Video Temporal Grounding [40.60371529725805]
  We propose an efficient preliminary in-domain fine-tuning paradigm for feature adaptation. We introduce Action-Cue-Injected Temporal Prompt Learning (ActPrompt), which injects action cues into the image encoder of VLM for better discovering action-sensitive patterns.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Open-Vocabulary Temporal Action Localization using Multimodal Guidance [67.09635853019005]
  OVTAL enables a model to recognize any desired action category in videos without the need to explicitly curate training data for all categories. This flexibility poses significant challenges, as the model must recognize not only the action categories seen during training but also novel categories specified at inference. We introduce OVFormer, a novel open-vocabulary framework extending ActionFormer with three key contributions.
  arXiv  Detail & Related papers  (2024-06-21T18:00:05Z)
• Test-Time Zero-Shot Temporal Action Localization [58.84919541314969]
  ZS-TAL seeks to identify and locate actions in untrimmed videos unseen during training. Training-based ZS-TAL approaches assume the availability of labeled data for supervised learning. We introduce a novel method that performs Test-Time adaptation for Temporal Action localization (T3AL)
  arXiv  Detail & Related papers  (2024-04-08T11:54:49Z)
• Harnessing Large Language Models for Training-free Video Anomaly Detection [34.76811491190446]
  Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Training-based methods are prone to be domain-specific, thus being costly for practical deployment. We propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm.
  arXiv  Detail & Related papers  (2024-04-01T09:34:55Z)
• Semi-supervised Active Learning for Video Action Detection [8.110693267550346]
  We develop a novel semi-supervised active learning approach which utilizes both labeled as well as unlabeled data. We evaluate the proposed approach on three different benchmark datasets, UCF-24-101, JHMDB-21, and Youtube-VOS.
  arXiv  Detail & Related papers  (2023-12-12T11:13:17Z)
• Dense Video Object Captioning from Disjoint Supervision [77.47084982558101]
  We propose a new task and model for dense video object captioning. This task unifies spatial and temporal localization in video. We show how our model improves upon a number of strong baselines for this new task.
  arXiv  Detail & Related papers  (2023-06-20T17:57:23Z)
• Active Learning with Effective Scoring Functions for Semi-Supervised Temporal Action Localization [15.031156121516211]
  This paper focuses on a rarely investigated yet practical task named semi-supervised TAL. We propose an effective active learning method, named AL-STAL. Experiment results show that AL-STAL outperforms the existing competitors and achieves satisfying performance compared with fully-supervised learning.
  arXiv  Detail & Related papers  (2022-08-31T13:39:38Z)
• Few-Shot Temporal Action Localization with Query Adaptive Transformer [105.84328176530303]
  TAL works rely on a large number of training videos with exhaustive segment-level annotation. Few-shot TAL aims to adapt a model to a new class represented by as few as a single video.
  arXiv  Detail & Related papers  (2021-10-20T13:18:01Z)
• Few-Shot Action Localization without Knowing Boundaries [9.959844922120523]
  We show that it is possible to learn to localize actions in untrimmed videos when only one/few trimmed examples of the target action are available at test time. We propose a network that learns to estimate Temporal Similarity Matrices (TSMs) that model a fine-grained similarity pattern between pairs of videos. Our method achieves performance comparable or better to state-of-the-art fully-supervised, few-shot learning methods.
  arXiv  Detail & Related papers  (2021-06-08T07:32:43Z)
• Weakly Supervised Temporal Action Localization Through Learning Explicit Subspaces for Action and Context [151.23835595907596]
  Methods learn to localize temporal starts and ends of action instances in a video under only video-level supervision. We introduce a framework that learns two feature subspaces respectively for actions and their context. The proposed approach outperforms state-of-the-art WS-TAL methods on three benchmarks.
  arXiv  Detail & Related papers  (2021-03-30T08:26:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.