Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

• URL: http://arxiv.org/abs/2604.18360v1
• Date: Mon, 20 Apr 2026 14:50:33 GMT
• Title: Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval
• Abstract summary: We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding.<n>We show OEA achieves comparable text-to-text retrieval performance to state-of-the-art M2D-CLAP.<n>Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-text retrieval performance to state-of-the-art M2D-CLAP.
• Score: 8.094136650811459
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-text retrieval systems based on Contrastive Language-Audio Pretraining (CLAP) achieve strong performance on traditional benchmarks; however, these benchmarks rely on caption-style queries that differ substantially from real-world search behavior, limiting their assessment of practical retrieval robustness. We present Omni-Embed-Audio (OEA), a retrieval-oriented encoder leveraging multimodal LLMs with native audio understanding. To systematically evaluate robustness beyond caption-style queries, we introduce User-Intent Queries (UIQs) - five formulations reflecting natural search behaviors: questions, commands, keyword tags, paraphrases, and exclusion-based negative queries. For negative queries, we develop a hard negative mining pipeline and propose discrimination metrics (HNSR, TFR) assessing models' ability to suppress acoustically similar distractors. Experiments on AudioCaps, Clotho, and MECAT show that OEA achieves comparable text-to-audio retrieval performance to state-of-the-art M2D-CLAP, while demonstrating clear advantages in two critical areas: (1) dominant text-to-text retrieval (+22% relative improvement), and (2) substantially superior hard negative discrimination (+4.3%p HNSR@10, +34.7% relative TFR@10), revealing that LLM backbones provide superior semantic understanding of complex queries.

Related papers

• Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB) [10.611072433458268]
  The Massive Sound Embedding Benchmark (MSEB) has emerged as a standard for evaluating the functional breadth of audio models.<n>The shift toward "audio-native" Large Language Models (LLMs) suggests a new paradigm where a single multimodal backbone may replace complex, task-specific pipelines.
  arXiv  Detail & Related papers  (2026-05-06T06:58:45Z)
• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation [97.0235251827591]
  Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs)<n>We study how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance.<n>Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance.
  arXiv  Detail & Related papers  (2026-03-19T17:50:07Z)
• AEQ-Bench: Measuring Empathy of Omni-Modal Large Models [55.722881748046895]
  We introduce AEQ-Bench, a novel benchmark to assess two core empathetic capabilities of omni-modal large models (OLMs)<n>AEQ-Bench incorporates two novel settings that vary in context specificity and speech tone.<n> Comprehensive assessment across linguistic and paralinguistic metrics reveals that OLMs trained with audio output capabilities generally outperformed models with text-only outputs.
  arXiv  Detail & Related papers  (2026-01-15T15:39:50Z)
• ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models [28.350243803500504]
  Large Audio Language Models (LALMs) extract and understand diverse information from audio.<n>LALMs are highly sensitive to how instructions are phrased, affecting instruction-following rates and task performance.<n>We introduce ISA-Bench, a benchmark evaluating instruction sensitivity for LALMs along three axes: instruction description, output format, and task composition.
  arXiv  Detail & Related papers  (2025-10-27T17:31:25Z)
• AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs [8.918587474371321]
  Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging.<n>We introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs.<n>Our system achieves a speedup of up to 127% over existing toolkits through optimized batch processing and parallel execution.
  arXiv  Detail & Related papers  (2025-09-09T15:30:40Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension [95.8442896569132]
  We introduce AIR-Bench, the first benchmark to evaluate the ability of Large Audio-Language Models (LALMs) to understand various types of audio signals and interact with humans in the textual format. Results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation.
  arXiv  Detail & Related papers  (2024-02-12T15:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.