Semi-Supervised Action Recognition with Temporal Contrastive Learning

• URL: http://arxiv.org/abs/2102.02751v1
• Date: Thu, 4 Feb 2021 17:28:35 GMT
• Title: Semi-Supervised Action Recognition with Temporal Contrastive Learning
• Authors: Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, Abir Das
• Abstract summary: We learn a two-pathway temporal contrastive model using unlabeled videos at two different speeds. We considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods.
• Score: 50.08957096801457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning to recognize actions from only a handful of labeled videos is a challenging problem due to the scarcity of tediously collected activity labels. We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action. Specifically, we propose to maximize the similarity between encoded representations of the same video at two different speeds as well as minimize the similarity between different videos played at different speeds. This way we use the rich supervisory information in terms of 'time' that is present in otherwise unsupervised pool of videos. With this simple yet effective strategy of manipulating video playback rates, we considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods across multiple diverse benchmark datasets and network architectures. Interestingly, our proposed approach benefits from out-of-domain unlabeled videos showing generalization and robustness. We also perform rigorous ablations and analysis to validate our approach.

Related papers

• SITAR: Semi-supervised Image Transformer for Action Recognition [20.609596080624662]
  This paper addresses video action recognition in a semi-supervised setting by leveraging only a handful of labeled videos. We capitalize on the vast pool of unlabeled samples and employ contrastive learning on the encoded super images. Our method demonstrates superior performance compared to existing state-of-the-art approaches for semi-supervised action recognition.
  arXiv  Detail & Related papers  (2024-09-04T17:49:54Z)
• TimeBalance: Temporally-Invariant and Temporally-Distinctive Video Representations for Semi-Supervised Action Recognition [68.53072549422775]
  We propose a student-teacher semi-supervised learning framework, TimeBalance. We distill the knowledge from a temporally-invariant and a temporally-distinctive teacher. Our method achieves state-of-the-art performance on three action recognition benchmarks.
  arXiv  Detail & Related papers  (2023-03-28T19:28:54Z)
• Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations [26.09611987412578]
  We introduce a new framework of contrastive action representation learning (CARL) to learn frame-wise action representation in a self-supervised or weakly-supervised manner. Specifically, we introduce a simple but effective video encoder that considers both spatial and temporal context. Our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification and even faster inference.
  arXiv  Detail & Related papers  (2022-12-06T16:42:22Z)
• Deep Video Prior for Video Consistency and Propagation [58.250209011891904]
  We present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. We show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior.
  arXiv  Detail & Related papers  (2022-01-27T16:38:52Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• Learning Implicit Temporal Alignment for Few-shot Video Classification [40.57508426481838]
  Few-shot video classification aims to learn new video categories with only a few labeled examples. It is particularly challenging to learn a class-invariant spatial-temporal representation in such a setting. We propose a novel matching-based few-shot learning strategy for video sequences in this work.
  arXiv  Detail & Related papers  (2021-05-11T07:18:57Z)
• Composable Augmentation Encoding for Video Representation Learning [94.2358972764708]
  We focus on contrastive methods for self-supervised video representation learning. A common paradigm in contrastive learning is to construct positive pairs by sampling different data views for the same instance, with different data instances as negatives. We propose an 'augmentation aware' contrastive learning framework, where we explicitly provide a sequence of augmentation parameterisations. We show that our method encodes valuable information about specified spatial or temporal augmentation, and in doing so also achieve state-of-the-art performance on a number of video benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T16:48:53Z)
• RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning [100.76672109782815]
  We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only. It is difficult to construct a suitable self-supervised task to well model both motion and appearance features. We propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels.
  arXiv  Detail & Related papers  (2020-10-27T16:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.