Exploiting Auxiliary Caption for Video Grounding

• URL: http://arxiv.org/abs/2301.05997v3
• Date: Sun, 24 Mar 2024 05:46:10 GMT
• Title: Exploiting Auxiliary Caption for Video Grounding
• Authors: Hongxiang Li, Meng Cao, Xuxin Cheng, Zhihong Zhu, Yaowei Li, Yuexian Zou,
• Abstract summary: Video grounding aims to locate a moment of interest matching a given query sentence from an untrimmed video. Previous works ignore the sparsity dilemma in video annotations, which fails to provide the context information between potential events and query sentences in the dataset. We propose an Auxiliary Caption Network (ACNet) for video grounding. Specifically, we first introduce dense video captioning to generate dense captions and then obtain auxiliary captions by Non-Auxiliary Caption Suppression (NACS) To capture the potential information in auxiliary captions, we propose Caption Guided Attention (CGA) project the semantic relations between auxiliary captions and
• Score: 66.77519356911051
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video. Previous works ignore the {sparsity dilemma} in video annotations, which fails to provide the context information between potential events and query sentences in the dataset. In this paper, we contend that exploiting easily available captions which describe general actions, i.e., auxiliary captions defined in our paper, will significantly boost the performance. To this end, we propose an Auxiliary Caption Network (ACNet) for video grounding. Specifically, we first introduce dense video captioning to generate dense captions and then obtain auxiliary captions by Non-Auxiliary Caption Suppression (NACS). To capture the potential information in auxiliary captions, we propose Caption Guided Attention (CGA) project the semantic relations between auxiliary captions and query sentences into temporal space and fuse them into visual representations. Considering the gap between auxiliary captions and ground truth, we propose Asymmetric Cross-modal Contrastive Learning (ACCL) for constructing more negative pairs to maximize cross-modal mutual information. Extensive experiments on three public datasets (i.e., ActivityNet Captions, TACoS and ActivityNet-CG) demonstrate that our method significantly outperforms state-of-the-art methods.

Related papers

• Video Summarization: Towards Entity-Aware Captions [75.71891605682931]
  We propose the task of summarizing news video directly to entity-aware captions. We show that our approach generalizes to existing news image captions dataset.
  arXiv  Detail & Related papers  (2023-12-01T23:56:00Z)
• Top-Down Framework for Weakly-supervised Grounded Image Captioning [19.00510117145054]
  Weakly-supervised grounded image captioning aims to generate the caption and ground (localize) predicted object words in the input image without using bounding box supervision. We propose a one-stage weakly-supervised grounded captioner that directly takes the RGB image as input to perform captioning and grounding at the top-down image level.
  arXiv  Detail & Related papers  (2023-06-13T01:42:18Z)
• Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos [57.830865926459914]
  We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments. Our framework is easily to tasks covering visually-grounded language understanding and generation.
  arXiv  Detail & Related papers  (2023-03-11T11:00:16Z)
• Tell me what you see: A zero-shot action recognition method based on natural language descriptions [3.136605193634262]
  We propose using video captioning methods to extract semantic information from videos. To the best of our knowledge, this is the first work to represent both videos and labels with descriptive sentences. We build a shared semantic space employing BERT-based embedders pre-trained in the paraphrasing task on multiple text datasets.
  arXiv  Detail & Related papers  (2021-12-18T17:44:07Z)
• Controllable Video Captioning with an Exemplar Sentence [89.78812365216983]
  We propose a novel Syntax Modulated Caption Generator (SMCG) incorporated in an encoder-decoder-reconstructor architecture. SMCG takes video semantic representation as an input, and conditionally modulates the gates and cells of long short-term memory network. We conduct experiments by collecting auxiliary exemplar sentences for two public video captioning datasets.
  arXiv  Detail & Related papers  (2021-12-02T09:24:45Z)
• Cross-Sentence Temporal and Semantic Relations in Video Activity Localisation [79.50868197788773]
  We develop a more accurate weakly-supervised solution by introducing Cross-Sentence Relations Mining. We explore two cross-sentence relational constraints: (1) trimmed ordering and (2) semantic consistency among sentences in a paragraph description of video activities. Experiments on two publicly available activity localisation datasets show the advantages of our approach over the state-of-the-art weakly supervised methods.
  arXiv  Detail & Related papers  (2021-07-23T20:04:01Z)
• Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos [63.94898634140878]
  Temporal language localization in videos aims to ground one video segment in an untrimmed video based on a given sentence query. We propose a Fine-grained Iterative Attention Network (FIAN) that consists of an iterative attention module for bilateral query-video in-formation extraction. We evaluate the proposed method on three challenging public benchmarks: Ac-tivityNet Captions, TACoS, and Charades-STA.
  arXiv  Detail & Related papers  (2020-08-06T04:09:03Z)
• OVC-Net: Object-Oriented Video Captioning with Temporal Graph and Detail Enhancement [44.228748086927375]
  We introduce the video-based object-oriented video captioning network (OVC)-Net via temporal graph and detail enhancement. To demonstrate the effectiveness, we conduct experiments on the new dataset and compare it with the state-of-the-art video captioning methods.
  arXiv  Detail & Related papers  (2020-03-08T04:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.