Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning

• URL: http://arxiv.org/abs/2603.11460v2
• Date: Fri, 13 Mar 2026 05:50:31 GMT
• Title: Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning
• Abstract summary: Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries.<n>The proposed framework, bfSTaRC, overcomes this limitation by supervising frame-level saliency through a highlight detection module.<n>We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics.
• Score: 7.051314153203061
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at https://github.com/ermitaju1/STaRC

Related papers

• Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization [69.67186845167568]
  Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events.<n>Existing methods primarily learn joint audio-visual representations in Euclidean space.<n>We propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework.
  arXiv  Detail & Related papers  (2026-06-05T08:23:58Z)
• Explainable Forensics of Manipulated Segments in Untrimmed Long Videos [50.190474724159465]
  We formulate the task of Temporal AI-Generated Segment localization and Explanation.<n>We introduce TASLE, a large-scale benchmark comprising 12,472 untrimmed videos with diverse manipulation patterns and rich annotation signals.<n>We propose MSLoc, a coarse-to-fine forensic baseline that combines a boundary-sensitive proposal generation module for efficient long-video scanning with an MLLM-based refinement module for precise boundary localization and interpretable reasoning.
  arXiv  Detail & Related papers  (2026-06-01T15:48:38Z)
• Foresee-to-Ground: From Predictive Temporal Perception to Evidence-Driven Reasoning for Video Temporal Grounding [86.37002814396674]
  We propose Foresee-to-Ground (F2G) for Video Temporal Grounding.<n>F2G integrates Predictive Temporal Perception with Evidence-Driven Reasoning.<n>It consistently improves grounding accuracy across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-05-21T04:03:25Z)
• Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection [24.57005070272218]
  Open-Vocabulary Temporal Action Detection (OV-TAD) aims to localize and classify action segments of unseen videos.<n> DFAlign is the first framework that leverages diffusion-based denoising to generate foreground knowledge for the guidance of action-video alignment.<n>Our method achieves state-of-the-art performance on two OV-TAD benchmarks.
  arXiv  Detail & Related papers  (2026-04-20T14:18:27Z)
• Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing [26.317163478761916]
  Weakly-supervised audio-visual video parsing seeks to detect audible, visible, and audio-visual events without temporal annotations.<n>We propose an exponential moving average (EMA)-guided pseudo supervision framework that generates reliable segment-level masks.<n>We also propose a class-aware cross-modal agreement (CMA) loss that aligns audio and visual embeddings at reliable segment-class pairs.
  arXiv  Detail & Related papers  (2025-09-17T15:38:05Z)
• TAG: A Simple Yet Effective Temporal-Aware Approach for Zero-Shot Video Temporal Grounding [12.57589714067771]
  Video Temporal Grounding aims to extract relevant video segments based on a given natural language query.<n>Previous approaches suffer from semantic fragmentation, where temporally continuous frames sharing the same semantics are split across multiple segments.<n>We propose a simple yet effective Temporal-Aware approach for zero-shot video temporal Grounding, which incorporates temporal pooling, temporal coherence clustering, and similarity adjustment.
  arXiv  Detail & Related papers  (2025-08-11T12:38:46Z)
• Local Compressed Video Stream Learning for Generic Event Boundary Detection [25.37983456118522]
  Event boundary detection aims to localize the generic, taxonomy-free event boundaries that segment videos into chunks. Existing methods typically require video frames to be decoded before feeding into the network. We propose a novel event boundary detection method that is fully end-to-end leveraging rich information in the compressed domain.
  arXiv  Detail & Related papers  (2023-09-27T06:49:40Z)
• Boosting Weakly-Supervised Temporal Action Localization with Text Information [94.48602948837664]
  We propose a Text-Segment Mining (TSM) mechanism, which constructs a text description based on the action class label, and regards the text as the query to mine all class-related segments. We also introduce a generative objective named Video-text Language Completion (VLC), which focuses on all semantic-related segments from videos to complete the text sentence. Surprisingly, we also find our proposed method can be seamlessly applied to existing methods, and improve their performances with a clear margin.
  arXiv  Detail & Related papers  (2023-05-01T00:07:09Z)
• Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding [148.46348699343991]
  Temporal language grounding aims to localize a video segment in an untrimmed video based on a natural language description. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. We propose a novel candidate-free framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG.
  arXiv  Detail & Related papers  (2022-10-21T13:10:27Z)
• Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding [78.71529237748018]
  Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance. We propose a commonsense-aware cross-modal alignment framework, which incorporates commonsense-guided visual and text representations into a complementary common space.
  arXiv  Detail & Related papers  (2022-04-04T13:07:05Z)
• Temporal Transductive Inference for Few-Shot Video Object Segmentation [27.140141181513425]
  Few-shot object segmentation (FS-VOS) aims at segmenting video frames using a few labelled examples of classes not seen during initial training. Key to our approach is the use of both global and local temporal constraints. Empirically, our model outperforms state-of-the-art meta-learning approaches in terms of mean intersection over union on YouTube-VIS by 2.8%.
  arXiv  Detail & Related papers  (2022-03-27T14:08:30Z)
• Implicit Motion Handling for Video Camouflaged Object Detection [60.98467179649398]
  We propose a new video camouflaged object detection (VCOD) framework. It can exploit both short-term and long-term temporal consistency to detect camouflaged objects from video frames.
  arXiv  Detail & Related papers  (2022-03-14T17:55:41Z)
• Reinforcement Learning for Weakly Supervised Temporal Grounding of Natural Language in Untrimmed Videos [134.78406021194985]
  We focus on the weakly supervised setting of this task that merely accesses to coarse video-level language description annotation without temporal boundary. We propose a emphBoundary Adaptive Refinement (BAR) framework that resorts to reinforcement learning to guide the process of progressively refining the temporal boundary.
  arXiv  Detail & Related papers  (2020-09-18T03:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.