Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning

• URL: http://arxiv.org/abs/2609.04183v1
• Date: Thu, 03 Sep 2026 17:58:02 GMT
• Title: Seeing Before Synthesizing: VLM-Guided Transition Event Discovery for Weakly-Supervised Dense Video Captioning
• Abstract summary: Weakly-Supervised Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video.<n>Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding.<n>We propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted.
• Score: 8.547456564908423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weakly-Supervised Dense Video Captioning aims to localize and describe multiple events in untrimmed videos given only an ordered set of event-level captions per video. Recent work synthesizes auxiliary transition captions via LLM to provide additional vision-language alignment, but these captions lack visual grounding and are rigidly assigned to every inter-event gap at a fixed location and duration. To address these, we propose Seeing Before Synthesizing (SBS), a framework that adaptively provides visually grounded linguistic guidance only where warranted. Leveraging a VLM, we generate frame-level narratives for the inter-event gaps and detect transitions from the semantic variation across them. For identified transitions, we then refine inter-event temporal masks by blending the temporal midpoint with the semantic change point and selecting the width that maximizes vision-language alignment. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance in both captioning and localization.

Related papers

• SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning [8.976074934042071]
  Weakly-Supervised Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries.<n>We propose SAIL, which constructs semantically-aware masks through cross-modal alignment.<n>Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions.
  arXiv  Detail & Related papers  (2026-03-05T17:59:58Z)
• TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchors [40.48528326378281]
  Dense video captioning aims to interpret and describe all temporally localized events throughout an input video.<n>Recent state-of-the-art methods leverage large language models (LLMs) to provide detailed moment descriptions for video data.<n>We propose TA-Prompting, which enhances VideoLLMs via Temporal Anchors that learn to precisely localize events and prompt the VideoLLMs to perform temporal-aware video event understanding.
  arXiv  Detail & Related papers  (2026-01-06T10:45:53Z)
• Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction [33.79474114703357]
  We propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI)<n>Our model consists of two core components: Cross-modal Frame Aggregation and Context-aware Feature Enhancement.<n>Experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.
  arXiv  Detail & Related papers  (2025-11-13T09:48:12Z)
• DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning [3.47287766500271]
  Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure.<n>We introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation.<n>We show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.
  arXiv  Detail & Related papers  (2025-10-27T22:29:08Z)
• Dense Video Captioning using Graph-based Sentence Summarization [80.52481563888459]
  We propose a graph-based partition-and-summarization framework for dense video captioning.<n>We focus on the summarization" stage, and propose a framework that effectively exploits the relationship between semantic words for summarization.
  arXiv  Detail & Related papers  (2025-06-25T16:23:43Z)
• Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization [83.7571144192515]
  We propose a division-and-summarization (DaS) framework for dense video captioning.<n>Considering that the generated sentences contain rich semantic descriptions, we formulate the dense video captioning task as a visual cue aided sentence summarization problem.<n>Our experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.
  arXiv  Detail & Related papers  (2025-06-25T16:02:04Z)
• FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation [55.01077993490845]
  Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling.<n>We introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework.
  arXiv  Detail & Related papers  (2025-06-20T07:46:40Z)
• Label-anticipated Event Disentanglement for Audio-Visual Video Parsing [61.08434062821899]
  We introduce a new decoding paradigm, underlinelabel sunderlineemunderlineantic-based underlineprojection (LEAP) LEAP works by iteratively projecting encoded latent features of audio/visual segments onto semantically independent label embeddings. To facilitate the LEAP paradigm, we propose a semantic-aware optimization strategy, which includes a novel audio-visual semantic similarity loss function.
  arXiv  Detail & Related papers  (2024-07-11T01:57:08Z)
• Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos [57.830865926459914]
  We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments. Our framework is easily to tasks covering visually-grounded language understanding and generation.
  arXiv  Detail & Related papers  (2023-03-11T11:00:16Z)
• DVCFlow: Modeling Information Flow Towards Human-like Video Captioning [163.71539565491113]
  Existing methods mainly generate captions from individual video segments, lacking adaptation to the global visual context. We introduce the concept of information flow to model the progressive information changing across video sequence and captions. Our method significantly outperforms competitive baselines, and generates more human-like text according to subject and objective tests.
  arXiv  Detail & Related papers  (2021-11-19T10:46:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.