Au-M-ol: A Unified Model for Medical Audio and Language Understanding

• URL: http://arxiv.org/abs/2604.23284v1
• Date: Sat, 25 Apr 2026 12:57:25 GMT
• Title: Au-M-ol: A Unified Model for Medical Audio and Language Understanding
• Abstract summary: Au-M-ol is a novel architecture that extends Large Language Models (LLMs) with audio processing.<n>It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR)<n>In experiments, Au-M-ol reduces Word Error Rate (WER) by 56% compared to state-of-the-art baselines on medical transcription tasks.
• Score: 0.9236074230806578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we present Au-M-ol, a novel multimodal architecture that extends Large Language Models (LLMs) with audio processing. It is designed to improve performance on clinically relevant tasks such as Automatic Speech Recognition (ASR). Au-M-ol has three main components: (1) an audio encoder that extracts rich acoustic features from medical speech, (2) an adaptation layer that maps audio features into the LLM input space, and (3) a pretrained LLM that performs transcription and clinical language understanding. This design allows the model to interpret spoken medical content directly, improving both accuracy and robustness. In experiments, Au-M-ol reduces Word Error Rate (WER) by 56\% compared to state-of-the-art baselines on medical transcription tasks. The model also performs well in challenging conditions, including noisy environments, domain-specific terminology, and speaker variability. These results suggest that Au-M-ol is a strong candidate for real-world clinical applications, where reliable and context-aware audio understanding is essential.

Related papers

• Qwen-Audio-3.0-ASR Technical Report [52.28849889799101]
  We present Qwen-Audio-3.0-ASR, a Mixture-of-Experts (MoE) LLM-based ASR system.<n>Qwen-Audio-3.0-ASR supports transcription across 30 languages and 16 Chinese dialectal varieties spanning eight major dialect regions.<n>We develop a dedicated streaming variant, Qwen-Audio-3.0-ASR-Streaming, for latency-sensitive applications.
  arXiv  Detail & Related papers  (2026-09-07T14:30:56Z)
• Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding [15.79973026677169]
  Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance.<n>We introduce AcuLa, a framework that instills semantic understanding into any audio encoder by aligning it with a medical language model.<n>Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools.
  arXiv  Detail & Related papers  (2025-12-04T14:30:58Z)
• fMRI-LM: Towards a Universal Foundation Model for Language-Aligned fMRI Understanding [30.02799445727658]
  We present fMRI-LM, a foundational model that bridges functional MRI (fMRI) and language through a three-stage framework.<n>In Stage 1, we learn a neural tokenizer that maps fMRI into discrete tokens embedded in a language-consistent space.<n>In Stage 2, a pretrained LLM is adapted to jointly model fMRI tokens and text, treating brain activity as a sequence that can be temporally predicted and linguistically described.<n>In Stage 3, we perform multi-task, multi-paradigm instruction tuning to endow fMRI-LM with high-level semantic understanding.
  arXiv  Detail & Related papers  (2025-11-24T20:26:59Z)
• Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations [18.74784108693223]
  Transformer-based speech language models (SLMs) have significantly improved neural speech recognition and understanding.<n>The extent to which SLMs encode nuanced syntactic and conceptual features remains unclear.<n>This study is the first to systematically evaluate the presence of contextual syntactic and semantic features across SLMs.
  arXiv  Detail & Related papers  (2025-09-19T06:29:33Z)
• Audio-Vision Contrastive Learning for Phonological Class Recognition [6.476789653980653]
  We propose a multimodal deep learning framework that combines real-time magnetic resonance imaging (rtMRI) and speech signals to classify three key articulatory dimensions.<n> Experimental results on the USC-TIMIT dataset show that our contrastive learning-based approach achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-07-23T16:44:22Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision [83.0622534215881]
  This work proposes an industry-level omni-modal large language model (LLM) pipeline that integrates auditory, visual, and linguistic modalities.<n>Our pipeline consists of three main components: First, a modular framework enabling flexible configuration of various encoder-LLM-decoder architectures.<n>Second, a lightweight training strategy that pre-trains audio-language alignment on the state-of-the-art vision-language model Qwen2.5-VL.<n>Third, an audio synthesis pipeline that generates high-quality audio-text data from diverse real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-26T17:26:36Z)
• Benchmarking Automatic Speech Recognition coupled LLM Modules for Medical Diagnostics [0.0]
  This report serves as my self project wherein models finetuned on medical call recordings are analysed.<n> Automatic Speech Recognition (ASR) for speech transcription and a Large Language Model (LLM) for context-aware, professional responses are analysed.
  arXiv  Detail & Related papers  (2025-02-18T14:05:13Z)
• Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time [73.7845280328535]
  We present Meerkat, an audio-visual LLM equipped with a fine-grained understanding of image and audio. Meerkat can tackle challenging tasks such as audio referred image grounding, image guided audio temporal localization, and audio-visual fact-checking. We achieve state-of-the-art performance on all these downstream tasks with a relative improvement of up to 37.12%.
  arXiv  Detail & Related papers  (2024-07-01T23:32:25Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)
• On decoder-only architecture for speech-to-text and large language model integration [59.49886892602309]
  Speech-LLaMA is a novel approach that effectively incorporates acoustic information into text-based large language models. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2023-07-08T06:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.