ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting

• URL: http://arxiv.org/abs/2504.15921v1
• Date: Tue, 22 Apr 2025 14:06:01 GMT
• Title: ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting
• Authors: Jian Hu, Dimitrios Korkinof, Shaogang Gong, Mariano Beguerisse-Diaz,
• Abstract summary: ViSMap is a system to summarise hour long videos with no-supervision.<n>We bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not)
• Score: 29.049727807251084
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce ViSMap: Unsupervised Video Summarisation by Meta Prompting, a system to summarise hour long videos with no-supervision. Most existing video understanding models work well on short videos of pre-segmented events, yet they struggle to summarise longer videos where relevant events are sparsely distributed and not pre-segmented. Moreover, long-form video understanding often relies on supervised hierarchical training that needs extensive annotations which are costly, slow and prone to inconsistency. With ViSMaP we bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not). We rely on LLMs to create optimised pseudo-summaries of long videos using segment descriptions from short ones. These pseudo-summaries are used as training data for a model that generates long-form video summaries, bypassing the need for expensive annotations of long videos. Specifically, we adopt a meta-prompting strategy to iteratively generate and refine creating pseudo-summaries of long videos. The strategy leverages short clip descriptions obtained from a supervised short video model to guide the summary. Each iteration uses three LLMs working in sequence: one to generate the pseudo-summary from clip descriptions, another to evaluate it, and a third to optimise the prompt of the generator. This iteration is necessary because the quality of the pseudo-summaries is highly dependent on the generator prompt, and varies widely among videos. We evaluate our summaries extensively on multiple datasets; our results show that ViSMaP achieves performance comparable to fully supervised state-of-the-art models while generalising across domains without sacrificing performance. Code will be released upon publication.

Related papers

• MR. Video: "MapReduce" is the Principle for Long Video Understanding [27.9561679446938]
  MR. Video is an agentic long video understanding framework. It performs detailed short video perception without being limited by context length. It achieves over 10% accuracy improvement on the challenging LVBench.
  arXiv  Detail & Related papers  (2025-04-22T17:59:41Z)
• Loong: Generating Minute-level Long Videos with Autoregressive Language Models [76.59124981781602]
  We propose Loong, a new autoregressive large language models (LLMs)-based video generator that can generate minute-long videos.<n>Specifically, we model the text tokens and video tokens as a unified sequence for autoregressive LLMs and train the model from scratch.<n>Our proposed Loong can be trained on 10-second videos and be extended to generate minute-level long videos conditioned on text prompts.
  arXiv  Detail & Related papers  (2024-10-03T17:59:02Z)
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs [20.168429351519055]
  Video understanding is a crucial next step for multimodal large language models (LMLMs) We propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities.
  arXiv  Detail & Related papers  (2024-06-13T17:50:05Z)
• Scaling Up Video Summarization Pretraining with Large Language Models [73.74662411006426]
  We introduce an automated and scalable pipeline for generating a large-scale video summarization dataset. We analyze the limitations of existing approaches and propose a new video summarization model that effectively addresses them. Our work also presents a new benchmark dataset that contains 1200 long videos each with high-quality summaries annotated by professionals.
  arXiv  Detail & Related papers  (2024-04-04T11:59:06Z)
• LongVLM: Efficient Long Video Understanding via Large Language Models [55.813206751150716]
  LongVLM is a simple yet powerful VideoLLM for long video understanding. We encode video representations that incorporate both local and global information. Our model produces more precise responses for long video understanding.
  arXiv  Detail & Related papers  (2024-04-04T11:33:29Z)
• VideoXum: Cross-modal Visual and Textural Summarization of Videos [54.0985975755278]
  We propose a new joint video and text summarization task. The goal is to generate both a shortened video clip along with the corresponding textual summary from a long video. The generated shortened video clip and text narratives should be semantically well aligned.
  arXiv  Detail & Related papers  (2023-03-21T17:51:23Z)
• HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
  HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
  arXiv  Detail & Related papers  (2023-01-05T21:53:19Z)
• TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency [133.75876535332003]
  We focus on summarizing instructional videos, an under-explored area of video summarization. Existing video summarization datasets rely on manual frame-level annotations. We propose an instructional video summarization network that combines a context-aware temporal video encoder and a segment scoring transformer.
  arXiv  Detail & Related papers  (2022-08-14T04:07:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.