Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

• URL: http://arxiv.org/abs/2604.12506v1
• Date: Tue, 14 Apr 2026 09:30:12 GMT
• Title: Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
• Abstract summary: Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion.<n>While excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception.<n>We propose Unified Audio (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components.
• Score: 53.41256816230278
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion: while excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception. We attribute this gap to a fundamental limitation of ASR-centric training, which provides precise linguistic targets but implicitly teaches models to suppress paralinguistic cues and acoustic events as noise. To address this, we propose Unified Audio Schema (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components -- Transcription, Paralinguistics, and Non-linguistic Events -- within a unified JSON format. This design achieves comprehensive acoustic coverage without sacrificing the tight audio-text alignment that enables reasoning. We validate the effectiveness of this supervision strategy by applying it to both discrete and continuous AudioLLM architectures. Extensive experiments on MMSU, MMAR, and MMAU demonstrate that UAS-Audio yields consistent improvements, boosting fine-grained perception by 10.9% on MMSU over the same-size state-of-the-art models while preserving robust reasoning capabilities. Our code and model are publicly available at https://github.com/Tencent/Unified_Audio_Schema.

Related papers

• UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception [53.41256816230278]
  UniAudio-Token is a framework that empowers semantic tokenizers with general audio perception without compromising speech ability.<n>SAP provide structured supervision by audio into linguistic content, vocal attributes, and auditory-scene primitives.<n>SAE introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers.
  arXiv  Detail & Related papers  (2026-05-29T16:36:21Z)
• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions [55.622295453533475]
  We introduce UniSonate, a unified flow-matching framework capable of synthesizing speech, music, and sound effects.<n>We propose a novel dynamic token injection mechanism that projects unstructured environmental sounds into a structured temporal latent space.<n>Experiments demonstrate that UniSonate achieves state-of-the-art performance in instruction-based TTS and TTM.
  arXiv  Detail & Related papers  (2026-04-24T04:26:04Z)
• Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens [62.56027815951259]
  Current audio language models are predominantly text-first, either extending pre-trained text LLM backbones or relying on semantic-only audio tokens.<n>This paper presents a systematic empirical study of native audio foundation models that apply next-token prediction to audio at scale.
  arXiv  Detail & Related papers  (2026-02-18T18:32:46Z)
• Eureka-Audio: Triggering Audio Intelligence in Compact Language Models [28.38037427018435]
  We present Eureka-Audio, a compact yet high-performance audio language model that achieves competitive performance against larger models.<n>Despite containing only 1.7B parameters, Eureka-Audio demonstrates strong performance on automatic speech recognition (ASR), audio understanding, and dense audio captioning.<n>To further enhance paralinguistic reasoning, we introduce DataFlux, a closed loop audio instruction data synthesis and verification pipeline.
  arXiv  Detail & Related papers  (2026-02-15T02:01:08Z)
• Pay More Attention To Audio: Mitigating Imbalance of Cross-Modal Attention in Large Audio Language Models [60.857389526958485]
  MATA is a training-free method that dynamically pushes LALMs to pay textbfMore textbfAttention textbfTo textbfAudio tokens within the self-attention mechanism.<n>Experiments on the MMAU and MMAR benchmarks confirm MATA's effectiveness, with consistent performance gains.
  arXiv  Detail & Related papers  (2025-09-23T09:02:15Z)
• MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning [9.580895202050947]
  Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL)<n>This work proposes a novel enhancement: integrating Multiple Choice Learning (MCL) to explicitly model prediction ambiguity and improve representation quality.
  arXiv  Detail & Related papers  (2025-08-18T08:10:07Z)
• Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models [91.11904427660043]
  We introduce Audio-Reasoner, a large-scale audio language model for deep reasoning in audio tasks.<n>We train Audio-Reasoner on CoTA, enabling it to achieve great logical capabilities in audio reasoning.<n>Our findings stress the core of structured CoT training in advancing audio reasoning.
  arXiv  Detail & Related papers  (2025-03-04T06:18:34Z)
• Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction [9.101978573666546]
  Baichuan-Audio is an end-to-end audio large language model that seamlessly integrates audio understanding and generation.<n>It features a text-guided aligned speech generation mechanism, enabling real-time speech interaction with both comprehension and generation capabilities.
  arXiv  Detail & Related papers  (2025-02-24T15:16:34Z)
• AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining [46.22290575167155]
  This paper proposes a framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework introduces a general representation of audio, called "language of audio" (LOA)
  arXiv  Detail & Related papers  (2023-08-10T17:55:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.