SITAR: Semi-supervised Image Transformer for Action Recognition

• URL: http://arxiv.org/abs/2409.02910v1
• Date: Wed, 4 Sep 2024 17:49:54 GMT
• Title: SITAR: Semi-supervised Image Transformer for Action Recognition
• Authors: Owais Iqbal, Omprakash Chakraborty, Aftab Hussain, Rameswar Panda, Abir Das,
• Abstract summary: This paper addresses video action recognition in a semi-supervised setting by leveraging only a handful of labeled videos. We capitalize on the vast pool of unlabeled samples and employ contrastive learning on the encoded super images. Our method demonstrates superior performance compared to existing state-of-the-art approaches for semi-supervised action recognition.
• Score: 20.609596080624662
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recognizing actions from a limited set of labeled videos remains a challenge as annotating visual data is not only tedious but also can be expensive due to classified nature. Moreover, handling spatio-temporal data using deep $3$D transformers for this can introduce significant computational complexity. In this paper, our objective is to address video action recognition in a semi-supervised setting by leveraging only a handful of labeled videos along with a collection of unlabeled videos in a compute efficient manner. Specifically, we rearrange multiple frames from the input videos in row-column form to construct super images. Subsequently, we capitalize on the vast pool of unlabeled samples and employ contrastive learning on the encoded super images. Our proposed approach employs two pathways to generate representations for temporally augmented super images originating from the same video. Specifically, we utilize a 2D image-transformer to generate representations and apply a contrastive loss function to minimize the similarity between representations from different videos while maximizing the representations of identical videos. Our method demonstrates superior performance compared to existing state-of-the-art approaches for semi-supervised action recognition across various benchmark datasets, all while significantly reducing computational costs.

Related papers

• Neuromorphic Synergy for Video Binarization [54.195375576583864]
  Bimodal objects serve as a visual form to embed information that can be easily recognized by vision systems. Neuromorphic cameras offer new capabilities for alleviating motion blur, but it is non-trivial to first de-blur and then binarize the images in a real-time manner. We propose an event-based binary reconstruction method that leverages the prior knowledge of the bimodal target's properties to perform inference independently in both event space and image space. We also develop an efficient integration method to propagate this binary image to high frame rate binary video.
  arXiv  Detail & Related papers  (2024-02-20T01:43:51Z)
• Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations [26.09611987412578]
  We introduce a new framework of contrastive action representation learning (CARL) to learn frame-wise action representation in a self-supervised or weakly-supervised manner. Specifically, we introduce a simple but effective video encoder that considers both spatial and temporal context. Our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification and even faster inference.
  arXiv  Detail & Related papers  (2022-12-06T16:42:22Z)
• Learning Trajectory-Aware Transformer for Video Super-Resolution [50.49396123016185]
  Video super-resolution aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Existing approaches usually align and aggregate video frames from limited adjacent frames. We propose a novel Transformer for Video Super-Resolution (TTVSR)
  arXiv  Detail & Related papers  (2022-04-08T03:37:39Z)
• Time-Equivariant Contrastive Video Representation Learning [47.50766781135863]
  We introduce a novel self-supervised contrastive learning method to learn representations from unlabelled videos. Our experiments show that time-equivariant representations achieve state-of-the-art results in video retrieval and action recognition benchmarks.
  arXiv  Detail & Related papers  (2021-12-07T10:45:43Z)
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning [40.556222166309524]
  We present SwinBERT, an end-to-end transformer-based model for video captioning. Our method adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input. Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames.
  arXiv  Detail & Related papers  (2021-11-25T18:02:12Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• A Video Is Worth Three Views: Trigeminal Transformers for Video-based Person Re-identification [77.08204941207985]
  Video-based person re-identification (Re-ID) aims to retrieve video sequences of the same person under non-overlapping cameras. We propose a novel framework named Trigeminal Transformers (TMT) for video-based person Re-ID.
  arXiv  Detail & Related papers  (2021-04-05T02:50:16Z)
• Composable Augmentation Encoding for Video Representation Learning [94.2358972764708]
  We focus on contrastive methods for self-supervised video representation learning. A common paradigm in contrastive learning is to construct positive pairs by sampling different data views for the same instance, with different data instances as negatives. We propose an 'augmentation aware' contrastive learning framework, where we explicitly provide a sequence of augmentation parameterisations. We show that our method encodes valuable information about specified spatial or temporal augmentation, and in doing so also achieve state-of-the-art performance on a number of video benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T16:48:53Z)
• Semi-Supervised Action Recognition with Temporal Contrastive Learning [50.08957096801457]
  We learn a two-pathway temporal contrastive model using unlabeled videos at two different speeds. We considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods.
  arXiv  Detail & Related papers  (2021-02-04T17:28:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.