GAID: Frame-Level Gated Audio-Visual Integration with Directional Perturbation for Text-Video Retrieval

• URL: http://arxiv.org/abs/2508.01711v1
• Date: Sun, 03 Aug 2025 10:44:24 GMT
• Title: GAID: Frame-Level Gated Audio-Visual Integration with Directional Perturbation for Text-Video Retrieval
• Abstract summary: GAID is a framework that integrates audio and visual features under textual guidance.<n>DASP injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference.<n>Experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results with notable efficiency gains.
• Score: 12.483734449829235
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-video retrieval requires precise alignment between language and temporally rich video signals. Existing methods predominantly exploit visual cues and often overlook complementary audio semantics or adopt coarse fusion strategies, leading to suboptimal multimodal representations. We present GAID, a framework that jointly address this gap via two key components: (i) a Frame-level Gated Fusion (FGF) that adaptively integrates audio and visual features under textual guidance, enabling fine-grained temporal alignment; and (ii) a Directional Adaptive Semantic Perturbation (DASP) that injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference. These modules complement each other -- fusion reduces modality gaps while perturbation regularizes cross-modal matching -- yielding more stable and expressive representations. Extensive experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results across all retrieval metrics with notable efficiency gains. Our code is available at https://github.com/YangBowenn/GAID.

Related papers

• Text-guided Feature Disentanglement for Cross-modal Gait Recognition [5.766298616867406]
  Gait recognition is a biometric technique that identifies individuals based on their walking patterns, offering advantages in long-range, non-intrusive scenarios.<n>Real-world scenarios often involve sensing modalities such as LiDAR and RGB cameras, making LiDAR-Cross-modal Gait recognition a critical yet challenging task.<n>To address this challenge, we propose TCFDNet, a Text-guided Cross-modal Feature Disentanglement Network.<n>Extensive experiments on SUSTech1K and FreeGait datasets demonstrate that TCFDNet achieves new state-of-the-art results and validate the effectiveness of the proposed modules.
  arXiv  Detail & Related papers  (2026-05-29T03:16:44Z)
• EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing [15.670119048865274]
  Weakly supervised Audio-Visual Video Parsing aims to recognize and temporally localize audio, visual, and audio-visual events in videos.<n>This paper proposes a novel framework that enhances uni-modal representations for both the pseudo-label generator and the AVVP model.
  arXiv  Detail & Related papers  (2026-05-09T06:13:27Z)
• PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition [9.145417637255651]
  Referring Audio-Visual (Ref-AVS) seeks to localize and segment target objects in video frames based on visual, auditory, and textual referring cues.<n>We propose PRIMED, inspired by the biased competition theory in cognitive neuroscience, which explicitly models both visual perception and language-driven prior modulation.
  arXiv  Detail & Related papers  (2026-05-08T02:40:34Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• Video-Level Language-Driven Video-Based Visible-Infrared Person Re-Identification [47.40091830500585]
  Video-based Visible-basedInfrared Person Re-Identification (VVIReID) aims to match pedestrian sequences across modalities by extracting modality-in sequence-level features.<n>A framework, video-level language-driven VVI-ReID (VLD), consists of two core modules: inmodality language (IMLP) and spatialtemporal aggregation.
  arXiv  Detail & Related papers  (2025-06-03T04:49:08Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap [38.5017989456818]
  DiffGAP is a novel approach incorporating a lightweight generative module within the contrastive space.<n>Our experimental results on VGGSound and AudioCaps datasets demonstrate that DiffGAP significantly improves performance in video/text-audio generation and retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-15T13:24:09Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)
• Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding [78.71529237748018]
  Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance. We propose a commonsense-aware cross-modal alignment framework, which incorporates commonsense-guided visual and text representations into a complementary common space.
  arXiv  Detail & Related papers  (2022-04-04T13:07:05Z)
• Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation [73.1652905564163]
  We address the problem of separating individual speech signals from videos using audio-visual neural processing. Most conventional approaches utilize frame-wise matching criteria to extract shared information between co-occurring audio and video. We propose a cross-modal affinity network (CaffNet) that learns global correspondence as well as locally-varying affinities between audio and visual streams.
  arXiv  Detail & Related papers  (2021-03-25T15:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.