Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences

• URL: http://arxiv.org/abs/2001.11122v3
• Date: Wed, 30 Sep 2020 17:17:06 GMT
• Title: Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
• Authors: Rosaura G. VidalMata, Walter J. Scheirer, Anna Kukleva, David Cox, Hilde Kuehne
• Abstract summary: This paper proposes an approach for the unsupervised learning of actions in untrimmed video sequences based on a joint visual-temporal embedding space. We show that the proposed approach is able to provide a meaningful visual and temporal embedding out of the visual cues present in contiguous video frames.
• Score: 25.299599341774204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding the structure of complex activities in untrimmed videos is a challenging task in the area of action recognition. One problem here is that this task usually requires a large amount of hand-annotated minute- or even hour-long video data, but annotating such data is very time consuming and can not easily be automated or scaled. To address this problem, this paper proposes an approach for the unsupervised learning of actions in untrimmed video sequences based on a joint visual-temporal embedding space. To this end, we combine a visual embedding based on a predictive U-Net architecture with a temporal continuous function. The resulting representation space allows detecting relevant action clusters based on their visual as well as their temporal appearance. The proposed method is evaluated on three standard benchmark datasets, Breakfast Actions, INRIA YouTube Instructional Videos, and 50 Salads. We show that the proposed approach is able to provide a meaningful visual and temporal embedding out of the visual cues present in contiguous video frames and is suitable for the task of unsupervised temporal segmentation of actions.

Related papers

• About Time: Advances, Challenges, and Outlooks of Action Understanding [57.76390141287026]
  This survey comprehensively reviews advances in uni- and multi-modal action understanding across a range of tasks. We focus on prevalent challenges, overview widely adopted datasets, and survey seminal works with an emphasis on recent advances.
  arXiv  Detail & Related papers  (2024-11-22T18:09:27Z)
• Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical video [10.287675722826028]
  This paper introduces Video Spatio-Temporal Disment Networks (VDST-Net) to disentangle information using semi-decoupled temporal knowledge distillation to predict high-quality class activation maps (CAMs) We demonstrate the efficacy of our framework on a public reference dataset and on a more challenging surgical video dataset where objects are, on average, present in less than 60% of annotated frames.
  arXiv  Detail & Related papers  (2024-07-22T16:52:32Z)
• What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions [55.574102714832456]
  spatial-temporal grounding describes the task of localizing events in space and time. Models for this task are usually trained with human-annotated sentences and bounding box supervision. We combine local representation learning, which focuses on fine-grained spatial information, with a global representation that captures higher-level representations.
  arXiv  Detail & Related papers  (2023-03-29T19:38:23Z)
• TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering [27.52568444236988]
  We propose an unsupervised approach for learning action classes from untrimmed video sequences. In particular, we propose a temporal embedding network that combines relative time prediction, feature reconstruction, and sequence-to-sequence learning. Based on the identified clusters, we decode the video into coherent temporal segments that correspond to semantically meaningful action classes.
  arXiv  Detail & Related papers  (2023-03-09T10:46:23Z)
• Video Salient Object Detection via Contrastive Features and Attention Modules [106.33219760012048]
  We propose a network with attention modules to learn contrastive features for video salient object detection. A co-attention formulation is utilized to combine the low-level and high-level features. We show that the proposed method requires less computation, and performs favorably against the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-11-03T17:40:32Z)
• Unsupervised Action Segmentation with Self-supervised Feature Learning and Co-occurrence Parsing [32.66011849112014]
  temporal action segmentation is a task to classify each frame in the video with an action label. In this work we explore a self-supervised method that operates on a corpus of unlabeled videos and predicts a likely set of temporal segments across the videos. We develop CAP, a novel co-occurrence action parsing algorithm that can not only capture the correlation among sub-actions underlying the structure of activities, but also estimate the temporal trajectory of the sub-actions in an accurate and general way.
  arXiv  Detail & Related papers  (2021-05-29T00:29:40Z)
• Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation [96.67525775629444]
  Action segmentation refers to inferring boundaries of semantically consistent visual concepts in videos. We present a fully automatic and unsupervised approach for segmenting actions in a video that does not require any training. Our proposal is an effective temporally-weighted hierarchical clustering algorithm that can group semantically consistent frames of the video.
  arXiv  Detail & Related papers  (2021-03-20T23:30:01Z)
• A Prospective Study on Sequence-Driven Temporal Sampling and Ego-Motion Compensation for Action Recognition in the EPIC-Kitchens Dataset [68.8204255655161]
  Action recognition is one of the top-challenging research fields in computer vision. ego-motion recorded sequences have become of important relevance. The proposed method aims to cope with it by estimating this ego-motion or camera motion.
  arXiv  Detail & Related papers  (2020-08-26T14:44:45Z)
• Hierarchical Attention Network for Action Segmentation [45.19890687786009]
  The temporal segmentation of events is an essential task and a precursor for the automatic recognition of human actions in the video. We propose a complete end-to-end supervised learning approach that can better learn relationships between actions over time. We evaluate our system on challenging public benchmark datasets, including MERL Shopping, 50 salads, and Georgia Tech Egocentric datasets.
  arXiv  Detail & Related papers  (2020-05-07T02:39:18Z)
• STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos [17.232631075144592]
  Methods for instance segmentation in videos typically follow the tracking-by-detection paradigm. We propose a novel approach that segments and tracks instances across space and time in a single stage. Our method achieves state-of-the-art results across multiple datasets and tasks.
  arXiv  Detail & Related papers  (2020-03-18T18:40:52Z)
• ZSTAD: Zero-Shot Temporal Activity Detection [107.63759089583382]
  We propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training can still be detected. We design an end-to-end deep network based on R-C3D as the architecture for this solution. Experiments on both the THUMOS14 and the Charades datasets show promising performance in terms of detecting unseen activities.
  arXiv  Detail & Related papers  (2020-03-12T02:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.