Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio

• URL: http://arxiv.org/abs/2606.23243v1
• Date: Mon, 22 Jun 2026 12:28:33 GMT
• Title: Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio
• Abstract summary: We propose Federated Self-Contextualization (FSC), a multimodal language model framework for in-context clinical audio diagnosis.<n>FSC constructs pseudo-label episodes via unsupervised clustering of audio representations, bypassing scarce real labels.<n>On held-out respiratory and cardiac conditions, FSC achieves 71.6% accuracy in 2-way 2-shot evaluation, outperforming audio-language baselines by over 9%.
• Score: 9.223303217530464
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical audio diagnosis in low-resource settings requires models that identify conditions from minimal examples without large annotated corpora. We propose Federated Self-Contextualization (FSC), a multimodal language model framework for in-context clinical audio diagnosis across federated hospital clients. FSC constructs pseudo-label episodes via unsupervised clustering of audio representations, bypassing scarce real diagnostic labels, and enables contextual reasoning from support-query pairs. Our progressive three-stage pipeline first aligns audio embeddings with the language model via caption-based pretraining, then adapts it for episodic in-context inference through federated optimization. At test time, given a small labeled support set, the model diagnoses an unseen query through multimodal reasoning. On held-out respiratory and cardiac conditions, FSC achieves 71.6% accuracy in 2-way 2-shot evaluation, outperforming audio-language baselines by over 9%.

Related papers

• When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition [10.779774155043166]
  We test whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech.<n>We find that diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate.<n>Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers.
  arXiv  Detail & Related papers  (2026-05-04T16:24:06Z)
• StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks [14.936669090239548]
  We present StethoLM, the first audio-language model specialized for cardiopulmonary auscultation.<n>It is capable of performing instruction-driven clinical tasks across the full spectrum of auscultation analysis.<n>Our work establishes a foundation for instruction-following AI systems in clinical auscultation.
  arXiv  Detail & Related papers  (2026-02-27T22:39:23Z)
• Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis [14.922065513695294]
  Resp-Agent is an autonomous multimodal system orchestrated by a novel Active Adrial Curriculum Agent (Thinker-A$2$CA)<n>To address the representation gap, we introduce a Modality-Weaving Diagnoser that weaves EHR data with audio tokens via Strategic Global Attention.<n>To address the data gap, we design a Flow Matching Generator that adapts a text-only Large Language Model (LLM) via modality injection.
  arXiv  Detail & Related papers  (2026-02-16T14:48:24Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• WESR: Scaling and Evaluating Word-level Event-Speech Recognition [59.21814194620928]
  Speech conveys not only linguistic information but also rich non-verbal vocal events such as laughing and crying.<n>We develop a refined taxonomy of 21 vocal events, with a new categorization into discrete (standalone) versus continuous (mixed with speech) types.<n>Based on the refined taxonomy, we introduce WESR-Bench, an expert-annotated evaluation set (900+ utterances) with a novel position-aware protocol.
  arXiv  Detail & Related papers  (2026-01-08T02:23:21Z)
• Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding [15.79973026677169]
  Pre-trained audio models excel at detecting acoustic patterns in auscultation sounds but often fail to grasp their clinical significance.<n>We introduce AcuLa, a framework that instills semantic understanding into any audio encoder by aligning it with a medical language model.<n>Our work demonstrates that this audio-language alignment transforms purely acoustic models into clinically-aware diagnostic tools.
  arXiv  Detail & Related papers  (2025-12-04T14:30:58Z)
• CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning [17.462121203082006]
  CaReAQA is an audio-language model that integrates a foundation audio model with the reasoning capabilities of large language models.<n>We introduce CaReSound, a benchmark dataset of annotated medical audio recordings enriched with metadata.<n> evaluation results show that CaReAQA achieves 86.2% accuracy on open-ended diagnostic reasoning tasks.
  arXiv  Detail & Related papers  (2025-05-02T11:42:46Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer [59.57249127943914]
  We present a multilingual Audio-Visual Speech Recognition model incorporating several enhancements to improve performance and audio noise robustness. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets. Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%.
  arXiv  Detail & Related papers  (2024-03-14T01:16:32Z)
• Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition [55.25565305101314]
  Articulatory features are invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition systems. This paper presents a cross-domain and cross-lingual A2A inversion approach that utilizes the parallel audio and ultrasound tongue imaging (UTI) data of the 24-hour TaL corpus in A2A model pre-training. Experiments conducted on three tasks suggested incorporating the generated articulatory features consistently outperformed the baseline TDNN and Conformer ASR systems.
  arXiv  Detail & Related papers  (2022-06-15T07:20:28Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.