Unlocking Speech Instruction Data Potential with Query Rewriting

• URL: http://arxiv.org/abs/2507.08603v1
• Date: Fri, 11 Jul 2025 13:55:45 GMT
• Title: Unlocking Speech Instruction Data Potential with Query Rewriting
• Authors: Yonghua Hei, Yibo Yan, Shuliang Liu, Huiyu Zhou, Linfeng Zhang, Xuming Hu,
• Abstract summary: End-to-end Large Speech Language Models(textbfLSLMs) demonstrate strong potential in response latency and speech comprehension capabilities.<n>However, the ability to follow speech instructions has not been fully realized due to the lack of datasets and heavily biased training tasks.<n>We propose a query rewriting framework with multi-LLM knowledge fusion, employing multiple agents to annotate and validate the synthesized speech.
• Score: 26.134056897363557
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: End-to-end Large Speech Language Models~(\textbf{LSLMs}) demonstrate strong potential in response latency and speech comprehension capabilities, showcasing general intelligence across speech understanding tasks. However, the ability to follow speech instructions has not been fully realized due to the lack of datasets and heavily biased training tasks. Leveraging the rich ASR datasets, previous approaches have used Large Language Models~(\textbf{LLMs}) to continue the linguistic information of speech to construct speech instruction datasets. Yet, due to the gap between LLM-generated results and real human responses, the continuation methods further amplify these shortcomings. Given the high costs of collecting and annotating speech instruction datasets by humans, using speech synthesis to construct large-scale speech instruction datasets has become a balanced and robust alternative. Although modern Text-To-Speech~(\textbf{TTS}) models have achieved near-human-level synthesis quality, it is challenging to appropriately convert out-of-distribution text instruction to speech due to the limitations of the training data distribution in TTS models. To address this issue, we propose a query rewriting framework with multi-LLM knowledge fusion, employing multiple agents to annotate and validate the synthesized speech, making it possible to construct high-quality speech instruction datasets without relying on human annotation. Experiments show that this method can transform text instructions into distributions more suitable for TTS models for speech synthesis through zero-shot rewriting, increasing data usability from 72\% to 93\%. It also demonstrates unique advantages in rewriting tasks that require complex knowledge and context-related abilities.

Related papers

• ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models [70.56468982313834]
  We propose ProsodyLM, which introduces a simple tokenization scheme amenable to learning prosody.<n>We find that ProsodyLM can learn surprisingly diverse emerging prosody processing capabilities through pre-training alone.
  arXiv  Detail & Related papers  (2025-07-27T00:59:01Z)
• Speechless: Speech Instruction Training Without Speech for Low Resource Languages [14.223895501862811]
  A scarcity of speech instruction data is essential for fine-tuning large language models to understand and execute spoken commands.<n>Our novel approach addresses this challenge by halting synthesis at the semantic representation level, bypassing the need for TTS.<n>We achieve this by aligning synthetic semantic representations with the pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text instructions while maintaining the ability to understand spoken instructions during inference.
  arXiv  Detail & Related papers  (2025-05-23T03:05:47Z)
• Scaling Speech-Text Pre-training with Synthetic Interleaved Data [31.77653849518526]
  Speech language models (SpeechLMs) accept speech input and produce speech output, allowing for more natural human-computer interaction.<n>Traditional approaches for developing SpeechLMs are constrained by the limited availability of unsupervised speech data and parallel speech-text data.<n>We propose a novel approach to scaling speech-text pre-training by leveraging large-scale synthetic interleaved data derived from text corpora.
  arXiv  Detail & Related papers  (2024-11-26T17:19:09Z)
• VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning [64.56272011710735]
  We propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the large language models (LLMs) backbone.<n>Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T00:36:06Z)
• Self-Powered LLM Modality Expansion for Large Speech-Text Models [62.27700381806554]
  Large language models (LLMs) exhibit remarkable performance across diverse tasks. This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning. We introduce a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning.
  arXiv  Detail & Related papers  (2024-10-04T04:34:24Z)
• DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data [84.01401439030265]
  Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs)<n>We present a simple yet effective automatic process for creating speech-text pair data.<n>Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data.
  arXiv  Detail & Related papers  (2024-09-30T07:01:21Z)
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment [82.86363991170546]
  We propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities. Our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. These findings highlight the potential to reshape instruction-following SLMs by incorporating descriptive rich, speech captions.
  arXiv  Detail & Related papers  (2024-06-27T03:52:35Z)
• GRASS: Unified Generation Model for Speech-to-Semantic Tasks [7.044414457214718]
  We introduce a unified end-to-end (E2E) framework that generates target text conditioned on a task-related prompt for audio data. Our proposed model achieves state-of-the-art (SOTA) results on many benchmarks covering speech named entity recognition, speech sentiment analysis, speech question answering, and more. To facilitate future work on instruction fine-tuning for speech-to-semantic tasks, we release our instruction dataset and code.
  arXiv  Detail & Related papers  (2023-09-06T06:44:26Z)
• SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data [100.46303484627045]
  We propose a cross-modal Speech and Language Model (SpeechLM) to align speech and text pre-training with a pre-defined unified representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB.
  arXiv  Detail & Related papers  (2022-09-30T09:12:10Z)
• Guided-TTS:Text-to-Speech with Untranscribed Speech [22.548875263927396]
  We present Guided-TTS, a high-quality TTS model that learns to generate speech from untranscribed speech data. For text-to-speech synthesis, we guide the generative process of the unconditional DDPM via phoneme classification to produce mel-spectrograms.
  arXiv  Detail & Related papers  (2021-11-23T10:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.