3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio

• URL: http://arxiv.org/abs/2603.23126v1
• Date: Tue, 24 Mar 2026 12:23:10 GMT
• Title: 3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio
• Abstract summary: We present VIRST-Audio, a framework built upon a pretrainedOS model integrated with a vision-language architecture.<n>We convert input audio into text using an ASR module and perform segmentation using text-based supervision.<n>We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place.
• Score: 6.447274127678917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-based Referring Video Object Segmentation (ARVOS) requires grounding audio queries into pixel-level object masks over time, posing challenges in bridging acoustic signals with spatio-temporal visual representations. In this report, we present VIRST-Audio, a practical framework built upon a pretrained RVOS model integrated with a vision-language architecture. Instead of relying on audio-specific training, we convert input audio into text using an ASR module and perform segmentation using text-based supervision, enabling effective transfer from text-based reasoning to audio-driven scenarios. To improve robustness, we further incorporate an existence-aware gating mechanism that estimates whether the referred target object is present in the video and suppresses predictions when it is absent, reducing hallucinated masks and stabilizing segmentation behavior. We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place, demonstrating strong generalization and reliable performance in audio-based referring video segmentation.

Related papers

• 2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA [47.992210130090065]
  We present ASR-SaSa2VA, a resource-efficient framework for audio-guided video segmentation.<n>Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
  arXiv  Detail & Related papers  (2026-04-27T01:19:56Z)
• Revisiting Audio-Visual Segmentation with Vision-Centric Transformer [60.83798235788669]
  Audio-Visual (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal.<n>We propose a new Vision-Centric Transformer framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information.<n>Our framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset.
  arXiv  Detail & Related papers  (2025-06-30T08:40:36Z)
• Can Textual Semantics Mitigate Sounding Object Segmentation Preference? [10.368382203643739]
  We argue that audio lacks robust semantics compared to vision, resulting in weak audio guidance over the visual space. Motivated by the the fact that text modality is well explored and contains rich abstract semantics, we propose leveraging text cues from the visual scene to enhance audio guidance. Our method exhibits enhanced sensitivity to audio when aided by text cues, achieving highly competitive performance on all three subsets.
  arXiv  Detail & Related papers  (2024-07-15T17:45:20Z)
• Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition [29.414663568089292]
  Audio-visual speech recognition aims to transcribe human speech using both audio and video modalities. In this study, we strengthen the video features by learning three temporal dynamics in video data. We achieve the state-of-the-art performance on the LRS2 and LRS3 AVSR benchmarks for the noise-dominant settings.
  arXiv  Detail & Related papers  (2024-07-04T01:25:20Z)
• Language-Guided Audio-Visual Source Separation via Trimodal Consistency [64.0580750128049]
  A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform. We adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets.
  arXiv  Detail & Related papers  (2023-03-28T22:45:40Z)
• MAViL: Masked Audio-Video Learners [68.61844803682145]
  We present Masked Audio-Video learners (MAViL) to train audio-visual representations. Pre-training with MAViL enables the model to perform well in audio-visual classification and retrieval tasks. For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on benchmarks.
  arXiv  Detail & Related papers  (2022-12-15T18:59:59Z)
• AVATAR: Unconstrained Audiovisual Speech Recognition [75.17253531162608]
  We propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) trained end-to-end from spectrograms and full-frame RGB. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise. We also create a new, real-world test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.
  arXiv  Detail & Related papers  (2022-06-15T17:33:19Z)
• Self-Supervised Learning of Audio-Visual Objects from Video [108.77341357556668]
  We introduce a model that uses attention to localize and group sound sources, and optical flow to aggregate information over time. We demonstrate the effectiveness of the audio-visual object embeddings that our model learns by using them for four downstream speech-oriented tasks.
  arXiv  Detail & Related papers  (2020-08-10T16:18:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.