AV-Unified: A Unified Framework for Audio-visual Scene Understanding

• URL: http://arxiv.org/abs/2603.06530v1
• Date: Fri, 06 Mar 2026 18:16:30 GMT
• Title: AV-Unified: A Unified Framework for Audio-visual Scene Understanding
• Abstract summary: AV-Unified is a framework that enables joint learning across a wide range of audio-visual scene understanding tasks.<n>We unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens.<n>To overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module.
• Score: 37.966403423498456
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When humans perceive the world, they naturally integrate multiple audio-visual tasks within dynamic, real-world scenes. However, current works such as event localization, parsing, segmentation and question answering are mostly explored individually, making it challenging to comprehensively understand complex audio-visual scenes and explore inter-task relationships. Hence, we propose \textbf{AV-Unified}, a unified framework that enables joint learning across a wide range of audio-visual scene understanding tasks. AV-Unified standardizes the diverse input-output formats of each task and incorporates a multi-scale spatiotemporal perception network to effectively capture audio-visual associations. Specifically, we unify the inputs and outputs of all supported tasks by converting them into sequences of discrete tokens, establishing a shared representation that allows a single architecture to be trained jointly across heterogeneous varied datasets. Considering the varying temporal granularity of audio-visual events, a multi-scale temporal perception module is designed to capture key cues. Meanwhile, to overcome the lack of auditory supervision in the visual domain, we design a cross-modal guidance-based spatial perception module that models spatial audio-visual associations. Furthermore, task-specific text prompts are employed to enhance the model's adaptability and task-awareness. Extensive experiments on benchmark datasets (e.g., AVE, LLP, MUSIC-AVQA, VGG-SS and AVS) demonstrate the effectiveness of AV-Unified across temporal, spatial, and spatiotemporal tasks.

Related papers

• Empowering Long-form Omni-modal Understanding with Robust Audio Perception [48.23485436432239]
  AVDC (Audio-Visual Decoupled Captions) is a large-scale dataset designed to disentangle visual and auditory semantics.<n>AVDC captures both modality-specific nuances and complex cross-modalinteractions.<n>AVDC-QA-CoT is a Chain-of-Thought augmented question-answering dataset to foster audio-visualreasoning.
  arXiv  Detail & Related papers  (2026-07-11T13:04:24Z)
• Audio-Visual World Models: Towards Multisensory Imagination in Sight and Sound [5.591620304505415]
  This work presents the first formal framework for Audio-Visual World Models (AVWM)<n>It formulates multimodal environment simulation as a partially observable decision process with audio-visual observations, fine-grained actions, and task rewards.<n>We propose an Audio-Visual Conditional Transformer with a novel modality expert architecture that balances visual and auditory learning.
  arXiv  Detail & Related papers  (2025-11-30T13:11:56Z)
• Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models [13.63552417613795]
  We propose a novel zero-shot AVS framework that eliminates task-specific training by leveraging multiple pretrained models.<n>Our approach integrates audio, vision, and text representations to bridge modality gaps, enabling precise sound source segmentation without AVS-specific annotations.
  arXiv  Detail & Related papers  (2025-06-06T21:06:35Z)
• AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation [62.682428307810525]
  We introduce AVS-Mamba, a selective state space model to address the audio-visual segmentation task.<n>Our framework incorporates two key components for video understanding and cross-modal learning.<n>Our approach achieves new state-of-the-art results on the AVSBench-object and AVS-semantic datasets.
  arXiv  Detail & Related papers  (2025-01-14T03:20:20Z)
• UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization [83.89550658314741]
  Video localization tasks aim to temporally locate specific instances in videos, including temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL) We present UniAV, a Unified Audio-Visual perception network, to achieve joint learning of TAL, SED and AVEL tasks for the first time.
  arXiv  Detail & Related papers  (2024-04-04T03:28:57Z)
• Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding [36.20990265600332]
  We introduce PU-VALOR, a comprehensive audio-visual dataset comprising over 114,000 pseudo-untrimmed videos with detailed temporal annotations.<n> PU-VALOR is derived from the large-scale but coarse-annotated audio-visual dataset VALOR, through a subtle method involving event-based video clustering.<n>We develop AVicuna, a model capable of aligning audio-visual events with temporal intervals and corresponding text tokens.
  arXiv  Detail & Related papers  (2024-03-24T19:50:49Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Leveraging Foundation models for Unsupervised Audio-Visual Segmentation [49.94366155560371]
  Audio-Visual (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. We introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training.
  arXiv  Detail & Related papers  (2023-09-13T05:05:47Z)
• Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline [53.07236039168652]
  We focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video. We introduce the first Untrimmed Audio-Visual dataset, which contains 10K untrimmed videos with over 30K audio-visual events. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audio-visual events with various lengths and capture dependencies between them in a single pass.
  arXiv  Detail & Related papers  (2023-03-22T22:00:17Z)
• Audio-Visual Fusion Layers for Event Type Aware Video Recognition [86.22811405685681]
  We propose a new model to address the multisensory integration problem with individual event-specific layers in a multi-task learning scheme. We show that our network is formulated with single labels, but it can output additional true multi-labels to represent the given videos.
  arXiv  Detail & Related papers  (2022-02-12T02:56:22Z)
• Learning Audio-Visual Correlations from Variational Cross-Modal Generation [35.07257471319274]
  We learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner. The learned correlations can be readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval.
  arXiv  Detail & Related papers  (2021-02-05T21:27:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.