Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning

• URL: http://arxiv.org/abs/2412.11467v1
• Date: Mon, 16 Dec 2024 05:48:44 GMT
• Title: Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning
• Abstract summary: This paper presents a dense video captioning network called Multi-Concept Cyclic Learning (MCCL)<n>We perform weakly supervised concept detection for each frame, and the detected concept embeddings are integrated into the video features to provide event cues.<n>In the captioning network, we establish a cyclic co-learning strategy where the generator guides the localizer for event localization through semantic matching.
• Score: 8.049305239825735
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dense video captioning aims to detect and describe all events in untrimmed videos. This paper presents a dense video captioning network called Multi-Concept Cyclic Learning (MCCL), which aims to: (1) detect multiple concepts at the frame level, using these concepts to enhance video features and provide temporal event cues; and (2) design cyclic co-learning between the generator and the localizer within the captioning network to promote semantic perception and event localization. Specifically, we perform weakly supervised concept detection for each frame, and the detected concept embeddings are integrated into the video features to provide event cues. Additionally, video-level concept contrastive learning is introduced to obtain more discriminative concept embeddings. In the captioning network, we establish a cyclic co-learning strategy where the generator guides the localizer for event localization through semantic matching, while the localizer enhances the generator's event semantic perception through location matching, making semantic perception and event localization mutually beneficial. MCCL achieves state-of-the-art performance on the ActivityNet Captions and YouCook2 datasets. Extensive experiments demonstrate its effectiveness and interpretability.

Related papers

• Knowledge-Refined Dual Context-Aware Network for Partially Relevant Video Retrieval [13.091084557639425]
  We present KDC-Net, a Knowledge-Refined Dual Context-Aware Network.<n>On the text side, a Hierarchical Semantic Aggregation module captures and adaptively fuses multi-scale phrase cues to enrich query semantics.<n>On the video side, a Dynamic Temporal Attention mechanism employs relative positional encoding and adaptive temporal windows to highlight key events with local temporal coherence.<n>Experiments on PRVR benchmarks show that KDC-Net consistently outperforms state-of-the-art methods, especially under low moment-to-video ratios.
  arXiv  Detail & Related papers  (2026-03-25T03:44:13Z)
• Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction [33.79474114703357]
  We propose an explicit temporal-semantic modeling framework called Context-Aware Cross-Modal Interaction (CACMI)<n>Our model consists of two core components: Cross-modal Frame Aggregation and Context-aware Feature Enhancement.<n>Experiments on the ActivityNet Captions and YouCook2 datasets demonstrate that CACMI achieves the state-of-the-art performance on dense video captioning task.
  arXiv  Detail & Related papers  (2025-11-13T09:48:12Z)
• Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization [83.7571144192515]
  We propose a division-and-summarization (DaS) framework for dense video captioning.<n>Considering that the generated sentences contain rich semantic descriptions, we formulate the dense video captioning task as a visual cue aided sentence summarization problem.<n>Our experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.
  arXiv  Detail & Related papers  (2025-06-25T16:02:04Z)
• PR-DETR: Injecting Position and Relation Prior for Dense Video Captioning [23.4119982709261]
  We propose a novel dense video captioning framework, named PR-DETR, which injects the explicit position and relation prior into the detection transformer.<n>On the one hand, we first generate a set of position-anchored queries to provide the scene-specific position and semantic information about potential events as position prior.<n>On the other hand, we further design an event relation encoder to explicitly calculate the relationship between event boundaries as relation prior to guide the event interaction to improve the semantic coherence of the captions.
  arXiv  Detail & Related papers  (2025-06-19T07:07:42Z)
• CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization [11.525177542345215]
  We introduce CACE-Net, which differs from most existing methods that solely use audio signals to guide visual information. We propose an audio-visual co-guidance attention mechanism that allows for adaptive bi-directional cross-modal attentional guidance. Experiments on the AVE dataset demonstrate that CACE-Net sets a new benchmark in the audio-visual event localization task.
  arXiv  Detail & Related papers  (2024-08-04T07:48:12Z)
• Event-aware Video Corpus Moment Retrieval [79.48249428428802]
  Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos. Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos. We propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval.
  arXiv  Detail & Related papers  (2024-02-21T06:55:20Z)
• Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos [57.830865926459914]
  We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments. Our framework is easily to tasks covering visually-grounded language understanding and generation.
  arXiv  Detail & Related papers  (2023-03-11T11:00:16Z)
• Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localization [8.530561069113716]
  We propose a novel video-level semantic consistency guidance network for the AVE localization task. It consists of two components: a cross-modal event representation extractor and an intra-modal semantic consistency enhancer. We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings.
  arXiv  Detail & Related papers  (2022-10-11T08:15:57Z)
• DVCFlow: Modeling Information Flow Towards Human-like Video Captioning [163.71539565491113]
  Existing methods mainly generate captions from individual video segments, lacking adaptation to the global visual context. We introduce the concept of information flow to model the progressive information changing across video sequence and captions. Our method significantly outperforms competitive baselines, and generates more human-like text according to subject and objective tests.
  arXiv  Detail & Related papers  (2021-11-19T10:46:45Z)
• Video Imprint [107.1365846180187]
  A new unified video analytics framework (ER3) is proposed for complex event retrieval, recognition and recounting. The proposed video imprint representation exploits temporal correlations among image features across video frames. The video imprint is fed into a reasoning network and a feature aggregation module, for event recognition/recounting and event retrieval tasks, respectively.
  arXiv  Detail & Related papers  (2021-06-07T00:32:47Z)
• Weakly-Supervised Multi-Level Attentional Reconstruction Network for Grounding Textual Queries in Videos [73.4504252917816]
  The task of temporally grounding textual queries in videos is to localize one video segment that semantically corresponds to the given query. Most of the existing approaches rely on segment-sentence pairs (temporal annotations) for training, which are usually unavailable in real-world scenarios. We present an effective weakly-supervised model, named as Multi-Level Attentional Reconstruction Network (MARN), which only relies on video-sentence pairs during the training stage.
  arXiv  Detail & Related papers  (2020-03-16T07:01:01Z)
• See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks [184.4379622593225]
  We introduce a novel network, called CO-attention Siamese Network (COSNet), to address the unsupervised video object segmentation task. We emphasize the importance of inherent correlation among video frames and incorporate a global co-attention mechanism. We propose a unified and end-to-end trainable framework where different co-attention variants can be derived for mining the rich context within videos.
  arXiv  Detail & Related papers  (2020-01-19T11:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.