Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

• URL: http://arxiv.org/abs/2605.12034v2
• Date: Wed, 13 May 2026 20:38:46 GMT
• Title: Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
• Abstract summary: We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration.<n>We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets.<n>We show that omni-modal progress is easier to interpret when evaluation controls visual leakage.
• Score: 20.125825685200432
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/

Related papers

• OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding [17.161087166882744]
  We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks.<n>It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities.<n> Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis.
  arXiv  Detail & Related papers  (2026-05-18T15:55:22Z)
• Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization [47.84878018101177]
  We introduce Omni-Persona, the first comprehensive benchmark for omnimodal personalization.<n>We formalize the task as cross-modal routing over the emphPersona Modality Graph, encompassing 4 task groups and 18 fine-grained tasks across $sim750$ items.
  arXiv  Detail & Related papers  (2026-05-11T05:11:22Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models [17.817469065260124]
  We introduce OmniACBench, a benchmark for evaluating context-grounded acoustic control in omni-modal models.<n>Given a spoken instruction, a text script, and an image, a model must read the script aloud with an appropriate tone and manner.<n>Experiments on eight models reveal their limitations in the proposed setting, despite their strong performance on prior textual-output evaluations.
  arXiv  Detail & Related papers  (2026-03-25T05:00:51Z)
• Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis [22.55861092515539]
  A critical bottleneck remains the lack of effective reward models (RMs)<n>We introduce textbf Omni-RRM, the first open-source rubric-grounded reward model.<n>It produces structured, multi-dimension preference judgments with dimension-wise justifications across textbftext, image, video, and audio
  arXiv  Detail & Related papers  (2026-01-31T18:20:45Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception [97.32606786622728]
  We present a systematic and comprehensive investigation of omni detailed perception from the perspectives of the data pipeline, models, and benchmark.<n>We propose Omni-Detective, an agentic data generation pipeline integrating tool-calling, to autonomously produce highly detailed yet minimally hallucinatory multimodal data.<n>Based on the data generated with Omni-Detective, we train two captioning models: Audio-Captioner for audio-only detailed perception, and Omni-Captioner for audio-visual detailed perception.
  arXiv  Detail & Related papers  (2025-10-14T17:00:09Z)
• Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision [83.0622534215881]
  This work proposes an industry-level omni-modal large language model (LLM) pipeline that integrates auditory, visual, and linguistic modalities.<n>Our pipeline consists of three main components: First, a modular framework enabling flexible configuration of various encoder-LLM-decoder architectures.<n>Second, a lightweight training strategy that pre-trains audio-language alignment on the state-of-the-art vision-language model Qwen2.5-VL.<n>Third, an audio synthesis pipeline that generates high-quality audio-text data from diverse real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-26T17:26:36Z)
• OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities [124.05360767047539]
  We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models. evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer.
  arXiv  Detail & Related papers  (2024-10-16T04:29:46Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.