Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings

• URL: http://arxiv.org/abs/2603.13362v1
• Date: Mon, 09 Mar 2026 10:29:47 GMT
• Title: Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings
• Abstract summary: Auscultation is a vital diagnostic tool, yet its utility is often limited by subjective interpretation.<n>We propose a framework that aligns multi-site auscultation recordings directly with a frozen Large Language Model (LLM) embedding space via gated cross-attention.
• Score: 7.307923185691307
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Auscultation is a vital diagnostic tool, yet its utility is often limited by subjective interpretation. While general-purpose Audio-Language Models (ALMs) excel in general domains, they struggle with the nuances of physiological signals. We propose a framework that aligns multi-site auscultation recordings directly with a frozen Large Language Model (LLM) embedding space via gated cross-attention. By leveraging the LLM's latent world knowledge, our approach moves beyond isolated classification toward holistic, patient-level assessment. On the CaReSound benchmark, our model achieves a state-of-the-art 0.865 F1-macro and 0.952 BERTScore. We demonstrate that lightweight, domain-specific encoders rival large-scale ALMs and that multi-site aggregation provides spatial redundancy that mitigates temporal truncation. This alignment of medical acoustics with text foundations offers a scalable path for bridging signal processing and clinical assessment.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis [51.81845420301345]
  SonoCLIP is a vision-language foundation model for fetal ultrasound analysis.<n>It integrates segmentation masks as mask-channel visual prompts within the vision encoder.<n>It achieves superior zero-shot transfer performance under both global and mask-guided inference.
  arXiv  Detail & Related papers  (2026-06-28T20:04:49Z)
• Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation [76.6507710204181]
  We present Echo-, an agentic multimodal reasoning model for ultrasound interpretation.<n> Echo- is trained to coordinate organ-specific detector outputs, integrate them with global visual context, and convert the resulting evidence into grounded diagnostic decisions.<n>Results suggest that agentic multimodal reasoning can turn specialized detectors into verifiable clinical evidence.
  arXiv  Detail & Related papers  (2026-04-30T15:31:00Z)
• Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis [4.619078510367921]
  Clinician Mimetic is a novel In-Context Learning (ICL) framework designed to synergize Discriminative Exemplar Coreset Selection (DECS) and Self-Refined Experience Summarization (SRES)<n> DECS simulates a clinician's ability to reference "anchor cases" by selecting discriminative visual coresets from noisy data at the computational level.<n>SRES mimics the cognition and reflection in clinical diagnosis by distilling diverse rollouts into a dynamic textual Experience Bank.
  arXiv  Detail & Related papers  (2026-03-29T15:29:59Z)
• OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis [53.01523944168442]
  Clinical interpretation relies on both slice-driven local features and volume-driven spatial representations.<n>Existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding.<n>We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios.
  arXiv  Detail & Related papers  (2026-02-18T00:42:41Z)
• Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis [14.922065513695294]
  Resp-Agent is an autonomous multimodal system orchestrated by a novel Active Adrial Curriculum Agent (Thinker-A$2$CA)<n>To address the representation gap, we introduce a Modality-Weaving Diagnoser that weaves EHR data with audio tokens via Strategic Global Attention.<n>To address the data gap, we design a Flow Matching Generator that adapts a text-only Large Language Model (LLM) via modality injection.
  arXiv  Detail & Related papers  (2026-02-16T14:48:24Z)
• Semi-Supervised Diseased Detection from Speech Dialogues with Multi-Level Data Modeling [27.224093715611534]
  We propose a novel framework for learning to detect medical conditions from speech acoustics.<n>Our end-to-end approach dynamically aggregates multi-granularity features and generates high-quality pseudo-labels.<n>This work provides a principled approach to learning from weak, far-end supervision in medical speech analysis.
  arXiv  Detail & Related papers  (2026-01-08T09:10:16Z)
• Explainable Multi-Modal Deep Learning for Automatic Detection of Lung Diseases from Respiratory Audio Signals [0.49581497240446293]
  This study presents an explainable multimodal deep learning framework for automatic lung-disease detection using respiratory audio signals.<n>The framework incorporates Grad-CAM, Integrated Gradients, and SHAP, generating interpretable spectral, temporal, and feature-level explanations.<n>The findings demonstrate the framework's potential for telemedicine, point-of-care diagnostics, and real-world respiratory screening.
  arXiv  Detail & Related papers  (2025-11-29T17:15:58Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• A Fully Open and Generalizable Foundation Model for Ultrasound Clinical Applications [77.3888788549565]
  We present EchoCare, a novel ultrasound foundation model for generalist clinical use.<n>We developed EchoCare via self-supervised learning on our curated, publicly available, large-scale dataset EchoCareData.<n>With minimal training, EchoCare outperforms state-of-the-art comparison models across 10 representative ultrasound benchmarks.
  arXiv  Detail & Related papers  (2025-09-15T10:05:31Z)
• Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation [4.286815457787583]
  We introduce Causal-SAM-LLM, a novel framework that elevates Large Language Models (LLMs) to the role of causal reasoners.<n>First, Linguistic Adrial Disentanglement (LAD) employs a Vision-Language Model to generate rich, textual descriptions of confounding image styles.<n>Second, Test-Time Causal Intervention (TCI) provides an interactive mechanism where an LLM interprets a clinician's natural language command to modulate the segmentation decoder's features in real-time.
  arXiv  Detail & Related papers  (2025-07-04T13:52:16Z)
• End-to-End Breast Cancer Radiotherapy Planning via LMMs with Consistency Embedding [47.360760580820966]
  We present RO-LMM, a comprehensive large multimodal model (LMM) tailored for the field of radiation oncology.<n>This model effectively manages a series of tasks within the clinical workflow, including clinical context summarization, radiation treatment plan suggestion, and plan-guided target volume segmentation.<n>We present a novel Consistency Embedding Fine-Tuning (CEFTune) technique, which boosts LMM's robustness to noisy inputs while preserving the consistency of handling clean inputs.
  arXiv  Detail & Related papers  (2023-11-27T14:49:06Z)
• Show from Tell: Audio-Visual Modelling in Clinical Settings [58.88175583465277]
  We consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference.
  arXiv  Detail & Related papers  (2023-10-25T08:55:48Z)
• Improving Multiple Sclerosis Lesion Segmentation Across Clinical Sites: A Federated Learning Approach with Noise-Resilient Training [75.40980802817349]
  Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. We introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites.
  arXiv  Detail & Related papers  (2023-08-31T00:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.