Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos

• URL: http://arxiv.org/abs/2005.02463v4
• Date: Sun, 18 Jul 2021 19:35:14 GMT
• Title: Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos
• Abstract summary: We present a self-supervised prediction framework capable of temporal event segmentation. We rely on predictions of high-level features computed by a standard deep learning backbone. We demonstrate our approach on video from 10 days (254 hours) of continuous wildlife monitoring data.
• Score: 6.279863832853343
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Using offline training schemes, researchers have tackled the event segmentation problem by providing full or weak-supervision through manually annotated labels or self-supervised epoch-based training. Most works consider videos that are at most 10's of minutes long. We present a self-supervised perceptual prediction framework capable of temporal event segmentation by building stable representations of objects over time and demonstrate it on long videos, spanning several days. The approach is deceptively simple but quite effective. We rely on predictions of high-level features computed by a standard deep learning backbone. For prediction, we use an LSTM, augmented with an attention mechanism, trained in a self-supervised manner using the prediction error. The self-learned attention maps effectively localize and track the event-related objects in each frame. The proposed approach does not require labels. It requires only a single pass through the video, with no separate training set. Given the lack of datasets of very long videos, we demonstrate our method on video from 10 days (254 hours) of continuous wildlife monitoring data that we had collected with required permissions. We find that the approach is robust to various environmental conditions such as day/night conditions, rain, sharp shadows, and windy conditions. For the task of temporally locating events, we had an 80% recall rate at 20% false-positive rate for frame-level segmentation. At the activity level, we had an 80% activity recall rate for one false activity detection every 50 minutes. We will make the dataset, which is the first of its kind, and the code available to the research community.

Related papers

• TTA-Vid: Generalized Test-Time Adaptation for Video Reasoning [54.70019148172847]
  We leverage the paradigm of Test-Time Reinforcement Learning on video-language data to allow for adapting a pretrained model to incoming video samples at test-time without explicit labels.<n>The proposed test-time adaptation for video approach (TTA-Vid) combines two components that work simultaneously.<n>TTA-Vid yields consistent improvements across various video reasoning tasks and is able to outperform current state-of-the-art methods trained on large-scale data.
  arXiv  Detail & Related papers  (2026-04-01T09:52:57Z)
• Open-World Skill Discovery from Unsegmented Demonstrations [21.57982251502703]
  We develop a self-supervised learning-based approach to segment online demonstration videos. We introduce Skill Boundary Detection (SBD), an annotation-free temporal video segmentation algorithm. Our method can leverage the diverse YouTube videos to train instruction-following agents.
  arXiv  Detail & Related papers  (2025-03-11T18:51:40Z)
• TimeLoc: A Unified End-to-End Framework for Precise Timestamp Localization in Long Videos [50.04992164981131]
  Temporal localization in untrimmed videos is crucial for video understanding but remains challenging. This task encompasses several subtasks, including temporal action localization, temporal video grounding, moment retrieval, and generic event boundary detection. We propose TimeLoc, a unified end-to-end framework for timestamp localization that can handle multiple tasks.
  arXiv  Detail & Related papers  (2025-03-09T09:11:26Z)
• Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild [30.99346916414286]
  We propose a novel paradigm to reduce annotation costs: pretraining the model on unlabeled, real-world videos. Direct pretraining on imperfect pseudo annotations presents significant challenges, including mismatched sentence-video pairs and imprecise temporal boundaries. We propose the ReCorrect algorithm, which comprises two main phases: semantics-guided refinement and memory-consensus correction.
  arXiv  Detail & Related papers  (2024-12-01T13:49:21Z)
• POPCat: Propagation of particles for complex annotation tasks [7.236620861573004]
  We propose a time efficient method called POPCat that exploits the multi-target and temporal features of video data. The method generates a semi-supervised pipeline for segmentation or box-based video annotation. The method shows a margin of improvement on recall/mAP50/mAP over the best results.
  arXiv  Detail & Related papers  (2024-06-24T23:43:08Z)
• TempNet: Temporal Attention Towards the Detection of Animal Behaviour in Videos [63.85815474157357]
  We propose an efficient computer vision- and deep learning-based method for the detection of biological behaviours in videos. TempNet uses an encoder bridge and residual blocks to maintain model performance with a two-staged, spatial, then temporal, encoder. We demonstrate its application to the detection of sablefish (Anoplopoma fimbria) startle events.
  arXiv  Detail & Related papers  (2022-11-17T23:55:12Z)
• Anomaly detection in surveillance videos using transformer based attention model [3.2968779106235586]
  This research suggests using a weakly supervised strategy to avoid annotating anomalous segments in training videos. The proposed framework is validated on real-world dataset i.e. ShanghaiTech Campus dataset.
  arXiv  Detail & Related papers  (2022-06-03T12:19:39Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• Unsupervised Action Segmentation with Self-supervised Feature Learning and Co-occurrence Parsing [32.66011849112014]
  temporal action segmentation is a task to classify each frame in the video with an action label. In this work we explore a self-supervised method that operates on a corpus of unlabeled videos and predicts a likely set of temporal segments across the videos. We develop CAP, a novel co-occurrence action parsing algorithm that can not only capture the correlation among sub-actions underlying the structure of activities, but also estimate the temporal trajectory of the sub-actions in an accurate and general way.
  arXiv  Detail & Related papers  (2021-05-29T00:29:40Z)
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning [60.720251418816815]
  We present a large-scale study on unsupervised representation learning from videos. Our objective encourages temporally-persistent features in the same video. We find that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds.
  arXiv  Detail & Related papers  (2021-04-29T17:59:53Z)
• Learning by Aligning Videos in Time [10.075645944474287]
  We present a self-supervised approach for learning video representations using temporal video alignment as a pretext task. We leverage a novel combination of temporal alignment loss and temporal regularization terms, which can be used as supervision signals for training an encoder network.
  arXiv  Detail & Related papers  (2021-03-31T17:55:52Z)
• Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation [96.67525775629444]
  Action segmentation refers to inferring boundaries of semantically consistent visual concepts in videos. We present a fully automatic and unsupervised approach for segmenting actions in a video that does not require any training. Our proposal is an effective temporally-weighted hierarchical clustering algorithm that can group semantically consistent frames of the video.
  arXiv  Detail & Related papers  (2021-03-20T23:30:01Z)
• Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos [82.02074241700728]
  In this paper, we present a prohibitive-level action recognition model that is trained with only video-frame labels. Our method per person detectors have been trained on large image datasets within Multiple Instance Learning framework. We show how we can apply our method in cases where the standard Multiple Instance Learning assumption, that each bag contains at least one instance with the specified label, is invalid.
  arXiv  Detail & Related papers  (2020-07-21T10:45:05Z)
• Revisiting Few-shot Activity Detection with Class Similarity Control [107.79338380065286]
  We present a framework for few-shot temporal activity detection based on proposal regression. Our model is end-to-end trainable, takes into account the frame rate differences between few-shot activities and untrimmed test videos, and can benefit from additional few-shot examples.
  arXiv  Detail & Related papers  (2020-03-31T22:02:38Z)
• SCT: Set Constrained Temporal Transformer for Set Supervised Action Segmentation [22.887397951846353]
  Weakly supervised approaches aim at learning temporal action segmentation from videos that are only weakly labeled. We propose an approach that can be trained end-to-end on such data. We evaluate our approach on three datasets where the approach achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2020-03-31T14:51:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.