Leveraging the Video-level Semantic Consistency of Event for
Audio-visual Event Localization
- URL: http://arxiv.org/abs/2210.05242v2
- Date: Fri, 20 Oct 2023 08:48:11 GMT
- Title: Leveraging the Video-level Semantic Consistency of Event for
Audio-visual Event Localization
- Authors: Yuanyuan Jiang, Jianqin Yin, Yonghao Dang
- Abstract summary: We propose a novel video-level semantic consistency guidance network for the AVE localization task.
It consists of two components: a cross-modal event representation extractor and an intra-modal semantic consistency enhancer.
We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings.
- Score: 8.530561069113716
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Audio-visual event (AVE) localization has attracted much attention in recent
years. Most existing methods are often limited to independently encoding and
classifying each video segment separated from the full video (which can be
regarded as the segment-level representations of events). However, they ignore
the semantic consistency of the event within the same full video (which can be
considered as the video-level representations of events). In contrast to
existing methods, we propose a novel video-level semantic consistency guidance
network for the AVE localization task. Specifically, we propose an event
semantic consistency modeling (ESCM) module to explore video-level semantic
information for semantic consistency modeling. It consists of two components: a
cross-modal event representation extractor (CERE) and an intra-modal semantic
consistency enhancer (ISCE). CERE is proposed to obtain the event semantic
information at the video level. Furthermore, ISCE takes video-level event
semantics as prior knowledge to guide the model to focus on the semantic
continuity of an event within each modality. Moreover, we propose a new
negative pair filter loss to encourage the network to filter out the irrelevant
segment pairs and a new smooth loss to further increase the gap between
different categories of events in the weakly-supervised setting. We perform
extensive experiments on the public AVE dataset and outperform the
state-of-the-art methods in both fully- and weakly-supervised settings, thus
verifying the effectiveness of our method.The code is available at
https://github.com/Bravo5542/VSCG.
Related papers
- Label-anticipated Event Disentanglement for Audio-Visual Video Parsing [61.08434062821899]
We introduce a new decoding paradigm, underlinelabel sunderlineemunderlineantic-based underlineprojection (LEAP)
LEAP works by iteratively projecting encoded latent features of audio/visual segments onto semantically independent label embeddings.
To facilitate the LEAP paradigm, we propose a semantic-aware optimization strategy, which includes a novel audio-visual semantic similarity loss function.
arXiv Detail & Related papers (2024-07-11T01:57:08Z) - Event-aware Video Corpus Moment Retrieval [79.48249428428802]
Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos.
Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos.
We propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval.
arXiv Detail & Related papers (2024-02-21T06:55:20Z) - Learning Grounded Vision-Language Representation for Versatile
Understanding in Untrimmed Videos [57.830865926459914]
We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events.
Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments.
Our framework is easily to tasks covering visually-grounded language understanding and generation.
arXiv Detail & Related papers (2023-03-11T11:00:16Z) - Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across
Modalities [43.048896440009784]
We propose the task of extracting event hierarchies from multimodal (video and text) data.
This reveals the structure of events and is critical to understanding them.
We show the limitations of state-of-the-art unimodal and multimodal baselines on this task.
arXiv Detail & Related papers (2022-06-14T23:24:15Z) - Multi-Modulation Network for Audio-Visual Event Localization [138.14529518908736]
We study the problem of localizing audio-visual events that are both audible and visible in a video.
Existing works focus on encoding and aligning audio and visual features at the segment level.
We propose a novel MultiModulation Network (M2N) to learn the above correlation and leverage it as semantic guidance.
arXiv Detail & Related papers (2021-08-26T13:11:48Z) - EAN: Event Adaptive Network for Enhanced Action Recognition [66.81780707955852]
We propose a unified action recognition framework to investigate the dynamic nature of video content.
First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events.
Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer.
arXiv Detail & Related papers (2021-07-22T15:57:18Z) - Learning Modality Interaction for Temporal Sentence Localization and
Event Captioning in Videos [76.21297023629589]
We propose a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos.
Our method turns out to achieve state-of-the-art performances on four standard benchmark datasets.
arXiv Detail & Related papers (2020-07-28T12:40:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.