Dense Video Captioning using Graph-based Sentence Summarization

• URL: http://arxiv.org/abs/2506.20583v1
• Date: Wed, 25 Jun 2025 16:23:43 GMT
• Title: Dense Video Captioning using Graph-based Sentence Summarization
• Authors: Zhiwang Zhang, Dong Xu, Wanli Ouyang, Luping Zhou,
• Abstract summary: We propose a graph-based partition-and-summarization framework for dense video captioning.<n>We focus on the summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization.
• Score: 80.52481563888459
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, dense video captioning has made attractive progress in detecting and captioning all events in a long untrimmed video. Despite promising results were achieved, most existing methods do not sufficiently explore the scene evolution within an event temporal proposal for captioning, and therefore perform less satisfactorily when the scenes and objects change over a relatively long proposal. To address this problem, we propose a graph-based partition-and-summarization (GPaS) framework for dense video captioning within two stages. For the ``partition" stage, a whole event proposal is split into short video segments for captioning at a finer level. For the ``summarization" stage, the generated sentences carrying rich description information for each segment are summarized into one sentence to describe the whole event. We particularly focus on the ``summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization. We achieve this goal by treating semantic words as nodes in a graph and learning their interactions by coupling Graph Convolutional Network (GCN) and Long Short Term Memory (LSTM), with the aid of visual cues. Two schemes of GCN-LSTM Interaction (GLI) modules are proposed for seamless integration of GCN and LSTM. The effectiveness of our approach is demonstrated via an extensive comparison with the state-of-the-arts methods on the two benchmarks ActivityNet Captions dataset and YouCook II dataset.

Related papers

• Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization [83.7571144192515]
  We propose a division-and-summarization (DaS) framework for dense video captioning.<n>Considering that the generated sentences contain rich semantic descriptions, we formulate the dense video captioning task as a visual cue aided sentence summarization problem.<n>Our experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.
  arXiv  Detail & Related papers  (2025-06-25T16:02:04Z)
• Fine-Grained Video Captioning through Scene Graph Consolidation [44.30028794237688]
  We propose a novel zero-shot video captioning approach that combines frame-level scene graphs from a video to obtain intermediate representations for caption generation.<n>Our method first generates frame-level captions using an image VLM, converts them into scene graphs, and consolidates these graphs to produce comprehensive video-level descriptions.
  arXiv  Detail & Related papers  (2025-02-23T03:59:05Z)
• Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos [57.830865926459914]
  We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments. Our framework is easily to tasks covering visually-grounded language understanding and generation.
  arXiv  Detail & Related papers  (2023-03-11T11:00:16Z)
• HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
  HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
  arXiv  Detail & Related papers  (2023-01-05T21:53:19Z)
• End-to-end Dense Video Captioning as Sequence Generation [83.90502354328679]
  We show how to model the two subtasks of dense video captioning jointly as one sequence generation task. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks integrated into large-scale pre-trained models.
  arXiv  Detail & Related papers  (2022-04-18T01:30:54Z)
• Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation [79.50868197788773]
  We develop a more accurate weakly-supervised solution by introducing Cross-Sentence Relations Mining. We explore two cross-sentence relational constraints: (1) trimmed ordering and (2) semantic consistency among sentences in a paragraph description of video activities. Experiments on two publicly available activity localisation datasets show the advantages of our approach over the state-of-the-art weakly supervised methods.
  arXiv  Detail & Related papers  (2021-07-23T20:04:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.