ALARM: Audio-Language Alignment for Reasoning Models

• URL: http://arxiv.org/abs/2603.09556v1
• Date: Tue, 10 Mar 2026 12:03:25 GMT
• Title: ALARM: Audio-Language Alignment for Reasoning Models
• Abstract summary: We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs.<n>For training, we construct a 6M-instance multi-task corpus spanning 19K hours of speech, music, and sound.<n>We achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.
• Score: 1.425194911075608
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.

Related papers

• JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions [66.71387365429062]
  JASTIN is an instruction-driven audio evaluation framework.<n>It formulates audio assessment as a self-instructed reasoning task.<n>It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks.
  arXiv  Detail & Related papers  (2026-05-06T05:18:42Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment [94.0709779805955]
  We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM)<n>It is designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning.<n>DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks.
  arXiv  Detail & Related papers  (2025-07-03T16:28:25Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs [33.12165044958361]
  Recent advances in Large Language Models (LLMs) show strong performance in speech recognition, including Audio-Visual Speech Recognition (AVSR)<n>To address this, we propose Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR.<n>Inspired by Matryoshka Representation Learning, our model encodes representations at multiple granularities with a single architecture.<n>For efficient fine-tuning, we introduce three LoRA-based strategies using global and scale-specific modules.
  arXiv  Detail & Related papers  (2025-03-09T00:02:10Z)
• AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? [65.49972312524724]
  multimodal large language models (MLLMs) have expanded their capabilities to include vision and audio modalities.<n>Our proposed DeafTest reveals that MLLMs often struggle with simple tasks humans find trivial.<n>We introduce AV-Odyssey Bench, a comprehensive audio-visual benchmark designed to assess whether those MLLMs can truly understand the audio-visual information.
  arXiv  Detail & Related papers  (2024-12-03T17:41:23Z)
• Large Language Models are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities.<n>We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities.<n>We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.79% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs [27.122094554340194]
  We extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities. The resulting end-to-end model, named AudioChatLlama, can utilize audio prompts as a replacement for text and sustain a conversation.
  arXiv  Detail & Related papers  (2023-11-12T06:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.