Scaling Open-Vocabulary Action Detection

• URL: http://arxiv.org/abs/2504.03096v2
• Date: Thu, 24 Apr 2025 18:07:13 GMT
• Title: Scaling Open-Vocabulary Action Detection
• Authors: Zhen Hao Sia, Yogesh Singh Rawat,
• Abstract summary: We introduce an encoder-only multimodal model for video action detection.<n>We exploit an existing closed-set action detection dataset for pretraining.<n>We devise a new benchmark to evaluate on existing closed-set action detection datasets without ever using them for training.
• Score: 3.1844358655583846
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this work, we focus on scaling open-vocabulary action detection. Existing approaches for action detection are predominantly limited to closed-set scenarios and rely on complex, parameter-heavy architectures. Extending these models to the open-vocabulary setting poses two key challenges: (1) the lack of large-scale datasets with many action classes for robust training, and (2) parameter-heavy adaptations to a pretrained vision-language contrastive model to convert it for detection, risking overfitting the additional non-pretrained parameters to base action classes. Firstly, we introduce an encoder-only multimodal model for video action detection, reducing the reliance on parameter-heavy additions for video action detection. Secondly, we introduce a simple weakly supervised training strategy to exploit an existing closed-set action detection dataset for pretraining. Finally, we depart from the ill-posed base-to-novel benchmark used by prior works in open-vocabulary action detection and devise a new benchmark to evaluate on existing closed-set action detection datasets without ever using them for training, showing novel results to serve as baselines for future work. Our code is available at: https://siatheindochinese.github.io/sia_act_page/

Related papers

• Open-Set Deepfake Detection: A Parameter-Efficient Adaptation Method with Forgery Style Mixture [58.60915132222421]
  We introduce an approach that is both general and parameter-efficient for face forgery detection. We design a forgery-style mixture formulation that augments the diversity of forgery source domains. We show that the designed model achieves state-of-the-art generalizability with significantly reduced trainable parameters.
  arXiv  Detail & Related papers  (2024-08-23T01:53:36Z)
• Open-Vocabulary Spatio-Temporal Action Detection [59.91046192096296]
  Open-vocabulary-temporal action detection (OV-STAD) is an important fine-grained video understanding task. OV-STAD requires training a model on a limited set of base classes with box and label supervision. To better adapt the holistic VLM for the fine-grained action detection task, we carefully fine-tune it on the localized video region-text pairs.
  arXiv  Detail & Related papers  (2024-05-17T14:52:47Z)
• Aligned Unsupervised Pretraining of Object Detectors with Self-training [41.03780087924593]
  Unsupervised pretraining of object detectors has recently become a key component of object detector training. We propose a framework that mitigates this issue and consists of three simple yet key ingredients. We show that our strategy is also capable of pretraining from scratch (including the backbone) and works on complex images like COCO.
  arXiv  Detail & Related papers  (2023-07-28T17:46:00Z)
• Meta-tuning Loss Functions and Data Augmentation for Few-shot Object Detection [7.262048441360132]
  Few-shot object detection is an emerging topic in the area of few-shot learning and object detection. We propose a training scheme that allows learning inductive biases that can boost few-shot detection. The proposed approach yields interpretable loss functions, as opposed to highly parametric and complex few-shot meta-models.
  arXiv  Detail & Related papers  (2023-04-24T15:14:16Z)
• DOAD: Decoupled One Stage Action Detection Network [77.14883592642782]
  Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. We present a decoupled one-stage network dubbed DOAD, to improve the efficiency for-temporal action detection.
  arXiv  Detail & Related papers  (2023-04-01T08:06:43Z)
• Open Set Action Recognition via Multi-Label Evidential Learning [25.15753429188536]
  We propose a new method for open set action recognition and novelty detection via MUlti-Label Evidential learning (MULE) Our Beta Evidential Neural Network estimates multi-action uncertainty with Beta densities based on actor-context-object relation representations. Our proposed approach achieves promising performance in single/multi-actor, single/multi-action settings.
  arXiv  Detail & Related papers  (2023-02-27T18:34:18Z)
• Self-supervised Pretraining with Classification Labels for Temporal Activity Detection [54.366236719520565]
  Temporal Activity Detection aims to predict activity classes per frame. Due to the expensive frame-level annotations required for detection, the scale of detection datasets is limited. This work proposes a novel self-supervised pretraining method for detection leveraging classification labels.
  arXiv  Detail & Related papers  (2021-11-26T18:59:28Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)
• One-Shot Object Detection without Fine-Tuning [62.39210447209698]
  We introduce a two-stage model consisting of a first stage Matching-FCOS network and a second stage Structure-Aware Relation Module. We also propose novel training strategies that effectively improve detection performance. Our method exceeds the state-of-the-art one-shot performance consistently on multiple datasets.
  arXiv  Detail & Related papers  (2020-05-08T01:59:23Z)
• Detection in Crowded Scenes: One Proposal, Multiple Predictions [79.28850977968833]
  We propose a proposal-based object detector, aiming at detecting highly-overlapped instances in crowded scenes. The key of our approach is to let each proposal predict a set of correlated instances rather than a single one in previous proposal-based frameworks. Our detector can obtain 4.9% AP gains on challenging CrowdHuman dataset and 1.0% $textMR-2$ improvements on CityPersons dataset.
  arXiv  Detail & Related papers  (2020-03-20T09:48:53Z)
• Incremental Few-Shot Object Detection [96.02543873402813]
  OpeN-ended Centre nEt is a detector for incrementally learning to detect class objects with few examples. ONCE fully respects the incremental learning paradigm, with novel class registration requiring only a single forward pass of few-shot training samples.
  arXiv  Detail & Related papers  (2020-03-10T12:56:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.