2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA

• URL: http://arxiv.org/abs/2604.23935v1
• Date: Mon, 27 Apr 2026 01:19:56 GMT
• Title: 2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA
• Abstract summary: We present ASR-SaSa2VA, a resource-efficient framework for audio-guided video segmentation.<n>Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
• Score: 47.992210130090065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-based video object segmentation aims to locate and segment objects in videos conditioned on audio cues, requiring precise understanding of both appearance and motion. Recent audio-driven video segmentation methods extend MLLMs by fusing audio and visual features for end-to-end localization. Despite their promise, these approaches are computationally intensive, struggle with aligning temporal audio cues to dynamic video content, and depend on large paired audio-video datasets. To address these challenges, we present ASR-SaSaSa2VA, a resource-efficient framework for audio-guided video segmentation. The key idea is to convert audio inputs into textual motion descriptions via automatic speech recognition (ASR) models and then leverage pre-trained text-based referring video segmentation models (e.g., SaSaSa2VA) for pixel-level predictions. To further enhance robustness, we incorporate a no-target expression detection module, implemented by a fine-tuned audio-based MLLM, which filters out audio clips that do not refer to any target object. This design allows the system to exploit strong pre-trained models while effectively handling ambiguous or irrelevant audio inputs. Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.

Related papers

• 3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio [6.447274127678917]
  We present VIRST-Audio, a framework built upon a pretrainedOS model integrated with a vision-language architecture.<n>We convert input audio into text using an ASR module and perform segmentation using text-based supervision.<n>We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place.
  arXiv  Detail & Related papers  (2026-03-24T12:23:10Z)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows [75.44753202066171]
  Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment.<n>We propose FoleyFlow to first align unimodal AV encoders via masked modeling training.<n>After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency.
  arXiv  Detail & Related papers  (2026-03-09T09:06:25Z)
• Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval [58.640807985155554]
  Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to a given query.<n>Most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality.<n>We propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR.
  arXiv  Detail & Related papers  (2025-08-06T09:58:43Z)
• SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation [50.03810359300705]
  SpA2V decomposes the generation process into two stages: audio-guided video planning and layout-grounded video generation.<n>We show that SpA2V excels in generating realistic videos with semantic and spatial alignment to the input audios.
  arXiv  Detail & Related papers  (2025-08-01T17:05:04Z)
• Hear-Your-Click: Interactive Object-Specific Video-to-Audio Generation [6.631248829195371]
  We introduce Hear-Your-Click, an interactive V2A framework enabling users to generate sounds for specific objects by clicking on the frame.<n>To achieve this, we propose Object-aware Contrastive Audio-Visual Fine-tuning (OCAV) with a Mask-guided Visual (MVE) to obtain object-level visual features aligned with audio.<n>To measure audio-visual correspondence, we designed a new evaluation metric, the CAV score.
  arXiv  Detail & Related papers  (2025-07-07T13:01:50Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)
• CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation [43.562848631392384]
  Audio-visual video segmentation aims to generate pixel-level maps of sound-producing objects within image frames. We propose a decoupled audio-video dependence combining audio and video features from their respective temporal and spatial dimensions.
  arXiv  Detail & Related papers  (2023-09-18T12:24:02Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.