Aligning Audio-Visual Joint Representations with an Agentic Workflow

• URL: http://arxiv.org/abs/2410.23230v2
• Date: Thu, 31 Oct 2024 04:20:22 GMT
• Title: Aligning Audio-Visual Joint Representations with an Agentic Workflow
• Authors: Shentong Mo, Yibing Song,
• Abstract summary: Visual content and accompanied audio signals naturally formulate a joint representation to improve audio-visual (AV) related applications. The importance of AV data alignment is usually undermined for achieving high-quality representation. We propose to improve AV joint representations from a data-centric perspective by aligning audio signals to visual data.
• Score: 34.4272246401837
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual content and accompanied audio signals naturally formulate a joint representation to improve audio-visual (AV) related applications. While studies develop various AV representation learning frameworks, the importance of AV data alignment is usually undermined for achieving high-quality representation. We observe that an audio signal may contain background noise interference. Also, non-synchronization may appear between audio and video streams. These non-strict data alignment limits representation quality and downgrade application performance. In this paper, we propose to improve AV joint representations from a data-centric perspective by aligning audio signals to visual data. Our alignment is conducted in an agentic workflow controlled by an LLM-based assistant named AVAgent. For each input AV data pair, our AVAgent uses a multi-modal LLM to convert audio and visual data into language descriptions separately (i.e., tool use). Then, AVAgent reasons whether this paired data is aligned well and plans to edit the audio signal if needed (i.e., planning). The audio editing is executed by predefined actions that filter noise or augment data. Moreover, we use a VLM to evaluate how modified audio signals match the visual content and provide feedback to AVAgent (i.e., reflection). The tool use, planning, and reflection steps operate cyclically to become an agentic workflow where audio signals are gradually aligned to visual content. To this end, existing methods can directly leverage the aligned AV data via our agentic workflow to improve AV joint representations. The experimental results comprehensively demonstrate the state-of-the-art performance of the proposed approach against previous baselines in diverse downstream tasks.

Related papers

• Learning Visual Affordance from Audio [29.90423475741895]
  We introduce Audio-Visual Affordance Grounding (AV-AG), a new task that segments object interaction regions from action sounds.<n>To support this task, we construct the first AV-AG dataset, comprising a large collection of action sounds, object images, and pixel-level affordance annotations.<n>We also propose AVAGFormer, a model equipped with a semantic-conditioned cross-modal mixer and a dual-head decoder.
  arXiv  Detail & Related papers  (2025-12-01T18:58:56Z)
• Revisiting Audio-Visual Segmentation with Vision-Centric Transformer [60.83798235788669]
  Audio-Visual (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal.<n>We propose a new Vision-Centric Transformer framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information.<n>Our framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset.
  arXiv  Detail & Related papers  (2025-06-30T08:40:36Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model [16.22530358172138]
  The framework implements a scoring mechanism for selecting aligned training data segments.<n>It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure.<n> Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture.
  arXiv  Detail & Related papers  (2025-03-12T09:48:38Z)
• Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition [39.206005299985605]
  We propose a novel GER paradigm for AVSR, termed AVGER, that follows the concept of listening and seeing again'' The proposed AVGER can reduce Word Error Rate (WER) by 24% compared to current mainstream AVSR systems.
  arXiv  Detail & Related papers  (2025-01-03T10:51:14Z)
• Relevance-guided Audio Visual Fusion for Video Saliency Prediction [23.873134951154704]
  We propose a novel relevance-guided audio-visual saliency prediction network dubbedSP. The Fusion module dynamically adjusts the retention of audio features based on the semantic relevance between audio and visual elements. The Multi-scale feature Synergy (MS) module integrates visual features from different encoding stages, enhancing the network's ability to represent objects at various scales.
  arXiv  Detail & Related papers  (2024-11-18T10:42:27Z)
• Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning [50.28566759231076]
  We propose an innovative, automatic approach to establish an audio dataset with high-quality captions. Specifically, we construct a large-scale, high-quality, audio-language dataset, named as Auto-ACD, comprising over 1.5M audio-text pairs. We employ LLM to paraphrase a congruent caption for each audio, guided by the extracted multi-modality clues.
  arXiv  Detail & Related papers  (2023-09-20T17:59:32Z)
• Text-to-feature diffusion for audio-visual few-shot learning [59.45164042078649]
  Few-shot learning from video data is a challenging and underexplored, yet much cheaper, setup. We introduce a unified audio-visual few-shot video classification benchmark on three datasets. We show that AV-DIFF obtains state-of-the-art performance on our proposed benchmark for audio-visual few-shot learning.
  arXiv  Detail & Related papers  (2023-09-07T17:30:36Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation [22.28510611697998]
  We propose a novel textbfAudio-aware query-enhanced textbfTRansformer (AuTR) to tackle the task. Unlike existing methods, our approach introduces a multimodal transformer architecture that enables deep fusion and aggregation of audio-visual features.
  arXiv  Detail & Related papers  (2023-07-25T03:59:04Z)
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding [57.08832099075793]
  Visually-guided sound source separation consists of three parts: visual feature extraction, multimodal feature fusion, and sound signal processing. This paper presents audio-visual predictive coding (AVPC) to tackle this task in parameter harmonizing and more effective manner. In addition, we develop a valid self-supervised learning strategy for AVPC via co-predicting two audio-visual representations of the same sound source.
  arXiv  Detail & Related papers  (2023-06-19T03:10:57Z)
• Transavs: End-To-End Audio-Visual Segmentation With Transformer [33.56539999875508]
  We propose TransAVS, the first Transformer-based end-to-end framework for Audio-Visual task. TransAVS disentangles the audio stream as audio queries, which will interact with images and decode into segmentation masks. Our experiments demonstrate that TransAVS achieves state-of-the-art results on the AVSBench dataset.
  arXiv  Detail & Related papers  (2023-05-12T03:31:04Z)
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations [88.30635799280923]
  We introduce AV-data2vec which builds audio-visual representations based on predicting contextualized representations. Results on LRS3 show that AV-data2vec consistently outperforms existing methods with the same amount of data and model size.
  arXiv  Detail & Related papers  (2023-02-10T02:55:52Z)
• Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition [29.05833230733178]
  We propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.
  arXiv  Detail & Related papers  (2022-07-13T08:07:19Z)
• Learning Representations from Audio-Visual Spatial Alignment [76.29670751012198]
  We introduce a novel self-supervised pretext task for learning representations from audio-visual content. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks.
  arXiv  Detail & Related papers  (2020-11-03T16:20:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.