MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge

• URL: http://arxiv.org/abs/2608.23234v1
• Date: Mon, 24 Aug 2026 13:22:28 GMT
• Title: MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge
• Abstract summary: The framework integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models.<n>We decompose the task into several stages and identify suitable foundation models for each stage.
• Score: 9.541493052844803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.

Related papers

• AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers [51.016506778535025]
  We develop a framework that generates high-quality audio-visual reasoning traces from single-modality teacher models.<n>We show that cross-modal training also transfers to single-modality tasks.
  arXiv  Detail & Related papers  (2026-04-17T18:13:27Z)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints [100.02131897927484]
  This paper focuses on the native training of Multimodal Large Language Models (MLLMs) in an end-to-end manner.<n>We propose a native MLLM called NaViL, combined with a simple and cost-effective recipe.<n> Experimental results on 14 multimodal benchmarks confirm the competitive performance of NaViL against existing MLLMs.
  arXiv  Detail & Related papers  (2025-10-09T17:59:37Z)
• SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes [30.870903750545004]
  We introduce a novel framework, termed SAM2-LOVE, which integrates textual, audio, and visual representations into a learnable token.<n>Technically, our approach includes a multimodal fusion module aimed at improving multimodal understanding of SAM2.<n>We conducted extensive experiments to demonstrate that SAM2-LOVE outperforms the SOTA by 8.5% in $calmathJ&F$ on the Ref-AVS benchmark.
  arXiv  Detail & Related papers  (2025-06-02T11:36:25Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• 4th PVUW MeViS 3rd Place Report: Sa2VA [105.88675577642204]
  We show that with a simple modification to the test time inference method on stronger MLLMs, we can lead to stronger results on MeVIS.<n>In particular, we adopt the recent method Sa2VA, a unified model for dense grounded understanding of both images and videos.
  arXiv  Detail & Related papers  (2025-04-01T07:06:47Z)
• Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time [73.7845280328535]
  We present Meerkat, an audio-visual LLM equipped with a fine-grained understanding of image and audio. Meerkat can tackle challenging tasks such as audio referred image grounding, image guided audio temporal localization, and audio-visual fact-checking. We achieve state-of-the-art performance on all these downstream tasks with a relative improvement of up to 37.12%.
  arXiv  Detail & Related papers  (2024-07-01T23:32:25Z)
• Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback [38.708690624594794]
  Video and text multimodal alignment remains challenging, primarily due to the deficient volume and quality of multimodal instruction-tune data. We present a novel alignment strategy that employs multimodal AI system to oversee itself called Reinforcement Learning from AI Feedback (RLAIF) In specific, we propose context-aware reward modeling by providing detailed video descriptions as context during the generation of preference feedback.
  arXiv  Detail & Related papers  (2024-02-06T06:27:40Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration [50.94902442781148]
  We propose a novel multi-modal large language model (LLM) that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. We construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances.
  arXiv  Detail & Related papers  (2023-06-15T12:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.