Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning

• URL: http://arxiv.org/abs/2603.11439v1
• Date: Thu, 12 Mar 2026 02:04:09 GMT
• Title: Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning
• Abstract summary: We propose utilizing role-specific queries that separate localization and captioning into independent components.<n>We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs.<n>We introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions.
• Score: 7.633598328818363
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.

Related papers

• Knowledge-Refined Dual Context-Aware Network for Partially Relevant Video Retrieval [13.091084557639425]
  We present KDC-Net, a Knowledge-Refined Dual Context-Aware Network.<n>On the text side, a Hierarchical Semantic Aggregation module captures and adaptively fuses multi-scale phrase cues to enrich query semantics.<n>On the video side, a Dynamic Temporal Attention mechanism employs relative positional encoding and adaptive temporal windows to highlight key events with local temporal coherence.<n>Experiments on PRVR benchmarks show that KDC-Net consistently outperforms state-of-the-art methods, especially under low moment-to-video ratios.
  arXiv  Detail & Related papers  (2026-03-25T03:44:13Z)
• Chain-of-Thought Textual Reasoning for Few-shot Temporal Action Localization [22.58434223222062]
  We propose a new few-shot temporal action localization method by Chain-of-Thought textual reasoning to improve localization performance.<n>Specifically, we design a novel few-shot learning framework that leverages textual semantic information to enhance the model's ability to capture action commonalities and variations.<n>We conduct extensive experiments on the publicly available ActivityNet1.3 and THUMOS14 datasets.
  arXiv  Detail & Related papers  (2025-04-18T04:35:35Z)
• Disentangle and denoise: Tackling context misalignment for video moment retrieval [16.939535169282262]
  Video Moment Retrieval aims to locate in-context video moments according to a natural language query. This paper proposes a cross-modal Context Denoising Network (CDNet) for accurate moment retrieval.
  arXiv  Detail & Related papers  (2024-08-14T15:00:27Z)
• MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval [53.417646562344906]
  Video Moment Retrieval (VMR) aims to localize a specific temporal segment within an untrimmed long video given a natural language query. Existing methods often suffer from inadequate training annotations, i.e., the sentence typically matches with a fraction of the prominent video content in the foreground with limited wording diversity. This intrinsic modality imbalance leaves a considerable portion of visual information remaining unaligned with text. In this work, we take an MLLM as a video narrator to generate plausible textual descriptions of the video, thereby mitigating the modality imbalance and boosting the temporal localization.
  arXiv  Detail & Related papers  (2024-06-25T18:39:43Z)
• Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement [72.7576395034068]
  Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content.
  arXiv  Detail & Related papers  (2024-02-21T07:16:06Z)
• MINOTAUR: Multi-task Video Grounding From Multimodal Queries [70.08973664126873]
  We present a single, unified model for tackling query-based video understanding in long-form videos. In particular, our model can address all three tasks of the Ego4D Episodic Memory benchmark.
  arXiv  Detail & Related papers  (2023-02-16T04:00:03Z)
• Hierarchical Deep Residual Reasoning for Temporal Moment Localization [48.108468456043994]
  We propose a Hierarchical Deep Residual Reasoning (HDRR) model, which decomposes the video and sentence into multi-level representations with different semantics. We also design the simple yet effective Res-BiGRUs for feature fusion, which is able to grasp the useful information in a self-adapting manner.
  arXiv  Detail & Related papers  (2021-10-31T07:13:34Z)
• Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos [55.52369116870822]
  This paper focuses on tackling the problem of temporal language localization in videos. It aims to identify the start and end points of a moment described by a natural language sentence in an untrimmed video.
  arXiv  Detail & Related papers  (2021-10-12T14:59:25Z)
• Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video [128.41392860714635]
  We introduce Weakly-Supervised Snoma-Temporally Detection (WSSTAD) in surveillance video. WSSTAD aims to localize a-temporal tube (i.e. sequence of bounding boxes at consecutive times) that encloses abnormal event. We propose a dual-branch network which takes as input proposals with multi-granularities in both spatial-temporal domains.
  arXiv  Detail & Related papers  (2021-08-09T06:11:14Z)
• Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos [76.21297023629589]
  We propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets.
  arXiv  Detail & Related papers  (2020-07-28T12:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.