INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

• URL: http://arxiv.org/abs/2608.16203v1
• Date: Mon, 17 Aug 2026 07:35:18 GMT
• Title: INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval
• Abstract summary: INSPIRE is the first benchmark for instruction-aware speech retrieval.<n>We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models.
• Score: 53.79335341504602
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing speech retrieval systems rely on fixed similarity matching and cannot adapt to diverse user intents. We introduce INSPIRE, the first benchmark for instruction-aware speech retrieval, in which natural-language instructions dynamically specify relevance criteria, including semantic content, speaker identity, speaking style, environmental sounds, and their combinations. We evaluate four retrieval paradigms: large audio-language models, cascaded pipelines, self-supervised speech models, and contrastive audio-language models. Our results reveal that no current method robustly handles all retrieval intents. Text-based approaches perform relatively better at semantic retrieval but struggle with paralinguistic attributes, while speech-based models are moderately better at capturing acoustic properties but falter at following instructions. These findings highlight the need for unified architectures capable of instruction-aware speech retrieval.

Related papers

• Quantifying the Generation Modality Gap in Speech-Text Language Models [22.8588553114451]
  Pure speech language models often lag behind text and speech-text language models in generating coherent content.<n>We study the speech-text modality gap in a family of spoken language models, based on flow matching for continuous acoustic feature generation.<n>We construct a unified generation-based evaluation suite that compares speech-only, text-only, and speech-text language models trained on matched data.
  arXiv  Detail & Related papers  (2026-09-13T19:09:07Z)
• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Expressive Speech Retrieval using Natural Language Descriptions of Speaking Style [13.415189715216354]
  We introduce the task of expressive speech retrieval.<n>The goal is to retrieve speech utterances spoken in a given style based on a natural language description of that style.<n>We train speech and text encoders to embed speech and text descriptions of speaking styles into a joint latent space.
  arXiv  Detail & Related papers  (2025-08-15T03:38:21Z)
• WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild [40.210224623581155]
  Multi-modal Large Language Models (LLMs) have demonstrated strong capabilities of direct speech interaction.<n>Existing evaluation methods often adapt text-based benchmarks, overlooking speech's unique characteristics and challenges.<n>Here, we introduce the first comprehensive benchmark designed to systematically evaluate end-to-end speechLLMs in practical speech conversations.
  arXiv  Detail & Related papers  (2025-06-27T03:18:45Z)
• SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation [10.828717295018123]
  We propose a unified embedding framework that eliminates the need for intermediate text representations.<n>Our model reduces pipeline latency by 50% while achieving higher retrieval accuracy compared to traditional two-stage methods.
  arXiv  Detail & Related papers  (2025-01-26T15:04:02Z)
• Classification of Spontaneous and Scripted Speech for Multilingual Audio [9.925703861731506]
  Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research.<n>This paper addresses the challenge of building a classifier that generalises well across different formats and languages.<n>We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers.
  arXiv  Detail & Related papers  (2024-12-16T15:45:10Z)
• Speechworthy Instruction-tuned Language Models [71.8586707840169]
  We show that both prompting and preference learning increase the speech-suitability of popular instruction-tuned LLMs. We share lexical, syntactical, and qualitative analyses to showcase how each method contributes to improving the speech-suitability of generated responses.
  arXiv  Detail & Related papers  (2024-09-23T02:34:42Z)
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment [82.86363991170546]
  We propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities. Our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. These findings highlight the potential to reshape instruction-following SLMs by incorporating descriptive rich, speech captions.
  arXiv  Detail & Related papers  (2024-06-27T03:52:35Z)
• Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue [71.15186328127409]
  Paralinguistics-enhanced Generative Pretrained Transformer (ParalinGPT) Model takes the conversational context of text, speech embeddings, and paralinguistic attributes as input prompts within a serialized multitasking framework. We utilize the Switchboard-1 corpus, including its sentiment labels as the paralinguistic attribute, as our spoken dialogue dataset.
  arXiv  Detail & Related papers  (2023-12-23T18:14:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.