Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

• URL: http://arxiv.org/abs/2104.13537v1
• Date: Wed, 28 Apr 2021 02:35:09 GMT
• Title: Shot Contrastive Self-Supervised Learning for Scene Boundary Detection
• Abstract summary: Shot contrastive learning approach (ShotCoL) learns a shot representation that maximizes the similarity between nearby shots. We show how to apply ShotCoL for the task of scene boundary detection to offer state-of-the-art performance on the MovieNet dataset. We take on the problem of finding timestamps in movies and TV episodes where video-ads can be inserted while offering a minimally disruptive viewing experience.
• Score: 7.4262338683231155
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Scenes play a crucial role in breaking the storyline of movies and TV episodes into semantically cohesive parts. However, given their complex temporal structure, finding scene boundaries can be a challenging task requiring large amounts of labeled training data. To address this challenge, we present a self-supervised shot contrastive learning approach (ShotCoL) to learn a shot representation that maximizes the similarity between nearby shots compared to randomly selected shots. We show how to apply our learned shot representation for the task of scene boundary detection to offer state-of-the-art performance on the MovieNet dataset while requiring only ~25% of the training labels, using 9x fewer model parameters and offering 7x faster runtime. To assess the effectiveness of ShotCoL on novel applications of scene boundary detection, we take on the problem of finding timestamps in movies and TV episodes where video-ads can be inserted while offering a minimally disruptive viewing experience. To this end, we collected a new dataset called AdCuepoints with 3,975 movies and TV episodes, 2.2 million shots and 19,119 minimally disruptive ad cue-point labels. We present a thorough empirical analysis on this dataset demonstrating the effectiveness of ShotCoL for ad cue-points detection.

Related papers

• Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding [67.1767666115821]
  This paper addresses the task of temporal sentence grounding (TSG)<n>In this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding.<n>Our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training.
  arXiv  Detail & Related papers  (2026-05-29T02:16:11Z)
• Multi-Modal Domain Adaptation Across Video Scenes for Temporal Video Grounding [59.599378814835205]
  Temporal Video Grounding (TVG) aims to localize the temporal boundary of a specific segment in an untrimmed video based on a given language query. We introduce a novel AMDA method to adaptively adjust the model's scene-related knowledge by incorporating insights from the target data.
  arXiv  Detail & Related papers  (2023-12-21T07:49:27Z)
• Towards Active Learning for Action Spotting in Association Football Videos [59.84375958757395]
  Analyzing football videos is challenging and requires identifying subtle and diverse-temporal patterns. Current algorithms face significant challenges when learning from limited annotated data. We propose an active learning framework that selects the most informative video samples to be annotated next.
  arXiv  Detail & Related papers  (2023-04-09T11:50:41Z)
• Scene Consistency Representation Learning for Video Scene Segmentation [26.790491577584366]
  We propose an effective Self-Supervised Learning (SSL) framework to learn better shot representations from long-term videos. We present an SSL scheme to achieve scene consistency, while exploring considerable data augmentation and shuffling methods to boost the model generalizability. Our method achieves the state-of-the-art performance on the task of Video Scene.
  arXiv  Detail & Related papers  (2022-05-11T13:31:15Z)
• Boundary-aware Self-supervised Learning for Video Scene Segmentation [20.713635723315527]
  Video scene segmentation is a task of temporally localizing scene boundaries in a video. We introduce three novel boundary-aware pretext tasks: Shot-Scene Matching, Contextual Group Matching and Pseudo-boundary Prediction. We achieve the new state-of-the-art on the MovieNet-SSeg benchmark.
  arXiv  Detail & Related papers  (2022-01-14T02:14:07Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• Toward Accurate Person-level Action Recognition in Videos of Crowded Scenes [131.9067467127761]
  We focus on improving the action recognition by fully-utilizing the information of scenes and collecting new data. Specifically, we adopt a strong human detector to detect spatial location of each frame. We then apply action recognition models to learn thetemporal information from video frames on both the HIE dataset and new data with diverse scenes from the internet.
  arXiv  Detail & Related papers  (2020-10-16T13:08:50Z)
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation [57.68890534164427]
  In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation. We simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. Our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks.
  arXiv  Detail & Related papers  (2020-05-20T18:00:05Z)
• Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos [6.279863832853343]
  We present a self-supervised prediction framework capable of temporal event segmentation. We rely on predictions of high-level features computed by a standard deep learning backbone. We demonstrate our approach on video from 10 days (254 hours) of continuous wildlife monitoring data.
  arXiv  Detail & Related papers  (2020-05-05T20:11:48Z)
• Event detection in coarsely annotated sports videos via parallel multi receptive field 1D convolutions [14.30009544149561]
  In problems such as sports video analytics, it is difficult to obtain accurate frame level annotations and exact event duration. We propose the task of event detection in coarsely annotated videos. We introduce a multi-tower temporal convolutional network architecture for the proposed task.
  arXiv  Detail & Related papers  (2020-04-13T19:51:25Z)
• Revisiting Few-shot Activity Detection with Class Similarity Control [107.79338380065286]
  We present a framework for few-shot temporal activity detection based on proposal regression. Our model is end-to-end trainable, takes into account the frame rate differences between few-shot activities and untrimmed test videos, and can benefit from additional few-shot examples.
  arXiv  Detail & Related papers  (2020-03-31T22:02:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.