Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization

• URL: http://arxiv.org/abs/2211.10056v1
• Date: Fri, 18 Nov 2022 07:01:28 GMT
• Title: Contrastive Losses Are Natural Criteria for Unsupervised Video Summarization
• Authors: Zongshang Pang, Yuta Nakashima, Mayu Otani, Hajime Nagahara
• Abstract summary: Video summarization aims to select the most informative subset of frames in a video to facilitate efficient video browsing. We propose three metrics featuring a desirable key frame: local dissimilarity, global consistency, and uniqueness. We show that by refining the pre-trained features with a lightweight contrastively learned projection module, the frame-level importance scores can be further improved.
• Score: 27.312423653997087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video summarization aims to select the most informative subset of frames in a video to facilitate efficient video browsing. Unsupervised methods usually rely on heuristic training objectives such as diversity and representativeness. However, such methods need to bootstrap the online-generated summaries to compute the objectives for importance score regression. We consider such a pipeline inefficient and seek to directly quantify the frame-level importance with the help of contrastive losses in the representation learning literature. Leveraging the contrastive losses, we propose three metrics featuring a desirable key frame: local dissimilarity, global consistency, and uniqueness. With features pre-trained on the image classification task, the metrics can already yield high-quality importance scores, demonstrating competitive or better performance than past heavily-trained methods. We show that by refining the pre-trained features with a lightweight contrastively learned projection module, the frame-level importance scores can be further improved, and the model can also leverage a large number of random videos and generalize to test videos with decent performance. Code available at https://github.com/pangzss/pytorch-CTVSUM.

Related papers

• Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs [56.040198387038025]
  We present a novel prompt-guided visual perception framework (abbreviated as Free Video-LLM) for efficient inference of training-free video LLMs. Our method effectively reduces the number of visual tokens while maintaining high performance across multiple video question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T12:35:12Z)
• Enhancing Video Summarization with Context Awareness [9.861215740353247]
  Video summarization automatically generate concise summaries by selecting techniques, shots, or segments that capture the video's essence. Despite the importance of video summarization, there is a lack of diverse and representative datasets. We propose an unsupervised approach that leverages video data structure and information for generating informative summaries.
  arXiv  Detail & Related papers  (2024-04-06T09:08:34Z)
• Learning from One Continuous Video Stream [70.30084026960819]
  We introduce a framework for online learning from a single continuous video stream. This poses great challenges given the high correlation between consecutive video frames. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation.
  arXiv  Detail & Related papers  (2023-12-01T14:03:30Z)
• Conditional Modeling Based Automatic Video Summarization [70.96973928590958]
  The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. A new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries.
  arXiv  Detail & Related papers  (2023-11-20T20:24:45Z)
• SELF-VS: Self-supervised Encoding Learning For Video Summarization [6.21295508577576]
  We propose a novel self-supervised video representation learning method using knowledge distillation to pre-train a transformer encoder. Our method matches its semantic video representation, which is constructed with respect to frame importance scores, to a representation derived from a CNN trained on video classification.
  arXiv  Detail & Related papers  (2023-03-28T14:08:05Z)
• Deep Unsupervised Key Frame Extraction for Efficient Video Classification [63.25852915237032]
  This work presents an unsupervised method to retrieve the key frames, which combines Convolutional Neural Network (CNN) and Temporal Segment Density Peaks Clustering (TSDPC) The proposed TSDPC is a generic and powerful framework and it has two advantages compared with previous works, one is that it can calculate the number of key frames automatically. Furthermore, a Long Short-Term Memory network (LSTM) is added on the top of the CNN to further elevate the performance of classification.
  arXiv  Detail & Related papers  (2022-11-12T20:45:35Z)
• NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition [89.84188594758588]
  A novel Non-saliency Suppression Network (NSNet) is proposed to suppress the responses of non-salient frames. NSNet achieves the state-of-the-art accuracy-efficiency trade-off and presents a significantly faster (2.44.3x) practical inference speed than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T09:41:22Z)
• Video Summarization Based on Video-text Modelling [0.0]
  We propose a multimodal self-supervised learning framework to obtain semantic representations of videos. We also introduce a progressive video summarization method, where the important content in a video is pinpointed progressively to generate better summaries. An objective evaluation framework is proposed to measure the quality of video summaries based on video classification.
  arXiv  Detail & Related papers  (2022-01-07T15:21:46Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• Unsupervised Video Summarization with a Convolutional Attentive Adversarial Network [32.90753137435032]
  We propose a convolutional attentive adversarial network (CAAN) to build a deep summarizer in an unsupervised way. Specifically, the generator employs a fully convolutional sequence network to extract global representation of a video, and an attention-based network to output normalized importance scores. The results show the superiority of our proposed method against other state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2021-05-24T07:24:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.