Stable Mean Teacher for Semi-supervised Video Action Detection

• URL: http://arxiv.org/abs/2412.07072v2
• Date: Mon, 23 Dec 2024 01:39:48 GMT
• Title: Stable Mean Teacher for Semi-supervised Video Action Detection
• Authors: Akash Kumar, Sirshapan Mitra, Yogesh Singh Rawat,
• Abstract summary: We focus on semi-supervised learning for video action detection. We present Stable Mean Teacher, a simple end-to-end teacher-based framework that benefits from improved and temporally consistent pseudo labels.
• Score: 3.5743998666556855
• License:
• Abstract: In this work, we focus on semi-supervised learning for video action detection. Video action detection requires spatiotemporal localization in addition to classification, and a limited amount of labels makes the model prone to unreliable predictions. We present Stable Mean Teacher, a simple end-to-end teacher-based framework that benefits from improved and temporally consistent pseudo labels. It relies on a novel Error Recovery (EoR) module, which learns from students' mistakes on labeled samples and transfers this knowledge to the teacher to improve pseudo labels for unlabeled samples. Moreover, existing spatiotemporal losses do not take temporal coherency into account and are prone to temporal inconsistencies. To address this, we present Difference of Pixels (DoP), a simple and novel constraint focused on temporal consistency, leading to coherent temporal detections. We evaluate our approach on four different spatiotemporal detection benchmarks: UCF101-24, JHMDB21, AVA, and YouTube-VOS. Our approach outperforms the supervised baselines for action detection by an average margin of 23.5% on UCF101-24, 16% on JHMDB21, and 3.3% on AVA. Using merely 10% and 20% of data, it provides competitive performance compared to the supervised baseline trained on 100% annotations on UCF101-24 and JHMDB21, respectively. We further evaluate its effectiveness on AVA for scaling to large-scale datasets and YouTube-VOS for video object segmentation, demonstrating its generalization capability to other tasks in the video domain. Code and models are publicly available.

Related papers

• Weakly Contrastive Learning via Batch Instance Discrimination and Feature Clustering for Small Sample SAR ATR [7.2932563202952725]
  We propose a novel framework named Batch Instance Discrimination and Feature Clustering (BIDFC) In this framework, embedding distance between samples should be moderate because of the high similarity between samples in the SAR images. Experimental results on the moving and stationary target acquisition and recognition (MSTAR) database indicate a 91.25% classification accuracy of our method fine-tuned on only 3.13% training data.
  arXiv  Detail & Related papers  (2024-08-07T08:39:33Z)
• HomE: Homography-Equivariant Video Representation Learning [62.89516761473129]
  We propose a novel method for representation learning of multi-view videos. Our method learns an implicit mapping between different views, culminating in a representation space that maintains the homography relationship between neighboring views. On action classification, our method obtains 96.4% 3-fold accuracy on the UCF101 dataset, better than most state-of-the-art self-supervised learning methods.
  arXiv  Detail & Related papers  (2023-06-02T15:37:43Z)
• Augment and Criticize: Exploring Informative Samples for Semi-Supervised Monocular 3D Object Detection [64.65563422852568]
  We improve the challenging monocular 3D object detection problem with a general semi-supervised framework. We introduce a novel, simple, yet effective Augment and Criticize' framework that explores abundant informative samples from unlabeled data. The two new detectors, dubbed 3DSeMo_DLE and 3DSeMo_FLEX, achieve state-of-the-art results with remarkable improvements for over 3.5% AP_3D/BEV (Easy) on KITTI.
  arXiv  Detail & Related papers  (2023-03-20T16:28:15Z)
• An Empirical Study of End-to-End Temporal Action Detection [82.64373812690127]
  Temporal action detection (TAD) is an important yet challenging task in video understanding. Rather than end-to-end learning, most existing methods adopt a head-only learning paradigm. We validate the advantage of end-to-end learning over head-only learning and observe up to 11% performance improvement.
  arXiv  Detail & Related papers  (2022-04-06T16:46:30Z)
• End-to-End Semi-Supervised Learning for Video Action Detection [23.042410033982193]
  We propose a simple end-to-end based approach effectively which utilizes the unlabeled data. Video action detection requires both, action class prediction as well as a-temporal consistency. We demonstrate the effectiveness of the proposed approach on two different action detection benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-08T18:11:25Z)
• Auxiliary Learning for Self-Supervised Video Representation via Similarity-based Knowledge Distillation [2.6519061087638014]
  We propose a novel approach to complement self-supervised pretraining via an auxiliary pretraining phase, based on knowledge similarity distillation, auxSKD. Our method deploys a teacher network that iteratively distils its knowledge to the student model by capturing the similarity information between segments of unlabelled video data. We also introduce a novel pretext task, Video Segment Pace Prediction or VSPP, which requires our model to predict the playback speed of a randomly selected segment of the input video to provide more reliable self-supervised representations.
  arXiv  Detail & Related papers  (2021-12-07T21:50:40Z)
• TCLR: Temporal Contrastive Learning for Video Representation [49.6637562402604]
  We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. With the commonly used 3D-ResNet-18 architecture, we achieve 82.4% (+5.1% increase over the previous best) top-1 accuracy on UCF101 and 52.9% (+5.4% increase) on HMDB51 action classification.
  arXiv  Detail & Related papers  (2021-01-20T05:38:16Z)
• Spatiotemporal Contrastive Video Representation Learning [87.56145031149869]
  We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn visual representations from unlabeled videos. Our representations are learned using a contrasttemporalive loss, where two augmented clips from the same short video are pulled together in the embedding space. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial.
  arXiv  Detail & Related papers  (2020-08-09T19:58:45Z)
• Self-supervised Temporal Discriminative Learning for Video Representation Learning [39.43942923911425]
  Temporal-discriminative features can hardly be extracted without using an annotated large-scale video action dataset for training. This paper proposes a novel Video-based Temporal-Discriminative Learning framework in self-supervised manner.
  arXiv  Detail & Related papers  (2020-08-05T13:36:59Z)
• Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos [82.02074241700728]
  In this paper, we present a prohibitive-level action recognition model that is trained with only video-frame labels. Our method per person detectors have been trained on large image datasets within Multiple Instance Learning framework. We show how we can apply our method in cases where the standard Multiple Instance Learning assumption, that each bag contains at least one instance with the specified label, is invalid.
  arXiv  Detail & Related papers  (2020-07-21T10:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.