Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization

• URL: http://arxiv.org/abs/2407.07024v2
• Date: Wed, 2 Oct 2024 10:37:58 GMT
• Title: Exploring Scalability of Self-Training for Open-Vocabulary Temporal Action Localization
• Authors: Jeongseok Hyun, Su Ho Han, Hyolim Kang, Joon-Young Lee, Seon Joo Kim,
• Abstract summary: We explore the scalability of self-training with unlabeled YouTube videos for temporal action localization (OV-TAL) Our approach consists of two stages: (1) a class-agnostic action localizer is trained on a human-labeled TAL dataset to generate pseudo-labels for unlabeled videos, and (2) the large-scale pseudo-labeled dataset is then used to train the localizer.
• Score: 44.00346424334902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The vocabulary size in temporal action localization (TAL) is limited by the scarcity of large-scale annotated datasets. To overcome this, recent works integrate vision-language models (VLMs), such as CLIP, for open-vocabulary TAL (OV-TAL). However, despite the success of VLMs trained on extensive datasets, existing OV-TAL methods still rely on human-labeled TAL datasets of limited size to train action localizers, limiting their generalizability. In this paper, we explore the scalability of self-training with unlabeled YouTube videos for OV-TAL. Our approach consists of two stages: (1) a class-agnostic action localizer is trained on a human-labeled TAL dataset to generate pseudo-labels for unlabeled videos, and (2) the large-scale pseudo-labeled dataset is then used to train the localizer. Extensive experiments demonstrate that leveraging web-scale videos in self-training significantly enhances the generalizability of an action localizer. Additionally, we identify limitations in existing OV-TAL evaluation schemes and propose a new benchmark for thorough assessment. Finally, we showcase the TAL performance of the large multimodal model Gemini-1.5 on our new benchmark. Code is released at https://github.com/HYUNJS/STOV-TAL.

Related papers

• UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark [20.15425745473231]
  Localizing unusual activities, such as human errors or surveillance incidents, in videos holds practical significance. To explore foundation models' capability in localizing unusual activity, we introduce UAL-Bench. UAL-Bench features three video datasets: UAG-OOPS, UAG- SSBD, UAG-FunQA, and an instruction-tune dataset: OOPS-UAG-Instruct. Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs.
  arXiv  Detail & Related papers  (2024-10-02T02:33:09Z)
• Open-Vocabulary Spatio-Temporal Action Detection [59.91046192096296]
  Open-vocabulary-temporal action detection (OV-STAD) is an important fine-grained video understanding task. OV-STAD requires training a model on a limited set of base classes with box and label supervision. To better adapt the holistic VLM for the fine-grained action detection task, we carefully fine-tune it on the localized video region-text pairs.
  arXiv  Detail & Related papers  (2024-05-17T14:52:47Z)
• Self-Supervised Learning for User Localization [8.529237718266042]
  Machine learning techniques have shown remarkable accuracy in localization tasks. Their dependency on vast amounts of labeled data, particularly Channel State Information (CSI) and corresponding coordinates, remains a bottleneck. We propose a pioneering approach that leverages self-supervised pretraining on unlabeled data to boost the performance of supervised learning for user localization based on CSI.
  arXiv  Detail & Related papers  (2024-04-19T21:49:10Z)
• Temporal Action Localization for Inertial-based Human Activity Recognition [9.948823510429902]
  Video-based Human Activity Recognition (TAL) has followed a segment-based prediction approach, localizing activity segments in a timeline of arbitrary length. This paper is the first to systematically demonstrate the applicability of state-of-the-art TAL models for both offline and near-online Human Activity Recognition (HAR) We show that by analyzing timelines as a whole, TAL models can produce more coherent segments and achieve higher NULL-class accuracy across all datasets.
  arXiv  Detail & Related papers  (2023-11-27T13:55:21Z)
• Optimization Efficient Open-World Visual Region Recognition [55.76437190434433]
  RegionSpot integrates position-aware localization knowledge from a localization foundation model with semantic information from a ViL model. Experiments in open-world object recognition show that our RegionSpot achieves significant performance gain over prior alternatives.
  arXiv  Detail & Related papers  (2023-11-02T16:31:49Z)
• A Global Model Approach to Robust Few-Shot SAR Automatic Target Recognition [6.260916845720537]
  It may not always be possible to collect hundreds of labeled samples per class for training deep learning-based SAR Automatic Target Recognition (ATR) models. This work specifically tackles the few-shot SAR ATR problem, where only a handful of labeled samples may be available to support the task of interest.
  arXiv  Detail & Related papers  (2023-03-20T00:24:05Z)
• Active Learning with Effective Scoring Functions for Semi-Supervised Temporal Action Localization [15.031156121516211]
  This paper focuses on a rarely investigated yet practical task named semi-supervised TAL. We propose an effective active learning method, named AL-STAL. Experiment results show that AL-STAL outperforms the existing competitors and achieves satisfying performance compared with fully-supervised learning.
  arXiv  Detail & Related papers  (2022-08-31T13:39:38Z)
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition [88.34182299496074]
  Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
  arXiv  Detail & Related papers  (2022-07-17T07:05:39Z)
• Reducing the Annotation Effort for Video Object Segmentation Datasets [50.893073670389164]
  densely labeling every frame with pixel masks does not scale to large datasets. We use a deep convolutional network to automatically create pseudo-labels on a pixel level from much cheaper bounding box annotations. We obtain the new TAO-VOS benchmark, which we make publicly available at www.vision.rwth-aachen.de/page/taovos.
  arXiv  Detail & Related papers  (2020-11-02T17:34:45Z)
• Improving Semantic Segmentation via Self-Training [75.07114899941095]
  We show that we can obtain state-of-the-art results using a semi-supervised approach, specifically a self-training paradigm. We first train a teacher model on labeled data, and then generate pseudo labels on a large set of unlabeled data. Our robust training framework can digest human-annotated and pseudo labels jointly and achieve top performances on Cityscapes, CamVid and KITTI datasets.
  arXiv  Detail & Related papers  (2020-04-30T17:09:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.