Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction

• URL: http://arxiv.org/abs/2511.10134v1
• Date: Fri, 14 Nov 2025 01:34:28 GMT
• Title: Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
• Authors: Mingda Jia, Weiliang Meng, Zenghuang Fu, Yiheng Li, Qi Zeng, Yifan Zhang, Ju Xin, Rongtao Xu, Jiguang Zhang, Xiaopeng Zhang,
• Abstract summary: We propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI)<n>Our model consists of two core components: Cross-modal Frame Aggregation and Context-aware Feature Enhancement.<n>Experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.
• Score: 33.79474114703357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dense video captioning jointly localizes and captions salient events in untrimmed videos. Recent methods primarily focus on leveraging additional prior knowledge and advanced multi-task architectures to achieve competitive performance. However, these pipelines rely on implicit modeling that uses frame-level or fragmented video features, failing to capture the temporal coherence across event sequences and comprehensive semantics within visual contexts. To address this, we propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI), which leverages both latent temporal characteristics within videos and linguistic semantics from text corpus. Specifically, our model consists of two core components: Cross-modal Frame Aggregation aggregates relevant frames to extract temporally coherent, event-aligned textual features through cross-modal retrieval; and Context-aware Feature Enhancement utilizes query-guided attention to integrate visual dynamics with pseudo-event semantics. Extensive experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.

Related papers

• DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning [3.47287766500271]
  Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure.<n>We introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation.<n>We show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.
  arXiv  Detail & Related papers  (2025-10-27T22:29:08Z)
• Temporal Grounding as a Learning Signal for Referring Video Object Segmentation [29.646697516547558]
  Referring Video Object (RVOS) aims to segment and track objects in videos based on natural language expressions, requiring precise alignment between visual content and textual queries.<n>Existing methods often suffer from semantic misalignment, largely due to indiscriminate frame sampling and supervision of all visible objects during training.<n>We introduce MeViS-M, a dataset built upon the challenging MeViS benchmark, where we manually annotate temporal spans when each object is referred to by the expression.
  arXiv  Detail & Related papers  (2025-08-16T07:34:43Z)
• MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval [53.417646562344906]
  Video Moment Retrieval (VMR) aims to localize a specific temporal segment within an untrimmed long video given a natural language query. Existing methods often suffer from inadequate training annotations, i.e., the sentence typically matches with a fraction of the prominent video content in the foreground with limited wording diversity. This intrinsic modality imbalance leaves a considerable portion of visual information remaining unaligned with text. In this work, we take an MLLM as a video narrator to generate plausible textual descriptions of the video, thereby mitigating the modality imbalance and boosting the temporal localization.
  arXiv  Detail & Related papers  (2024-06-25T18:39:43Z)
• SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation [35.063881868130075]
  This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. We propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin.
  arXiv  Detail & Related papers  (2023-05-26T15:13:44Z)
• Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding [112.3913646778859]
  We propose a simple yet effective video-language modeling framework, S-ViLM. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features. S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T22:45:07Z)
• Temporal Perceiving Video-Language Pre-training [112.1790287726804]
  This work introduces a novel text-video localization pre-text task to enable fine-grained temporal and semantic alignment. Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description. Our method connects the fine-grained frame representations with the word representations and implicitly distinguishes representations of different instances in the single modality.
  arXiv  Detail & Related papers  (2023-01-18T12:15:47Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)
• Visual Spatio-temporal Relation-enhanced Network for Cross-modal Text-Video Retrieval [17.443195531553474]
  Cross-modal retrieval of texts and videos aims to understand the correspondence between vision and language. We propose a Visual S-temporal Relation-enhanced semantic network (CNN-SRNet), a cross-temporal retrieval framework. Experiments are conducted on both MSR-VTT and MSVD datasets.
  arXiv  Detail & Related papers  (2021-10-29T08:23:40Z)
• Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos [55.52369116870822]
  This paper focuses on tackling the problem of temporal language localization in videos. It aims to identify the start and end points of a moment described by a natural language sentence in an untrimmed video.
  arXiv  Detail & Related papers  (2021-10-12T14:59:25Z)
• Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos [76.21297023629589]
  We propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos. Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets.
  arXiv  Detail & Related papers  (2020-07-28T12:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.