Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening

• URL: http://arxiv.org/abs/2606.14788v1
• Date: Wed, 10 Jun 2026 21:43:44 GMT
• Title: Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening
• Abstract summary: NeurMLLM is an efficient framework for neurodegenerative disease staging.<n>NeurMLLM encodes the spectrograms and Mel-frequency cepstral coefficients of audio data with vision transformers.<n>It projects their representations into the embedding space of a large language model.
• Score: 5.760251413371908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Voice-based screening offers a scalable and non-invasive way to assess neurodegenerative diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD), but their staging remains challenging due to the difficulty of integrating heterogeneous data. This paper presents NeurMLLM, an efficient multimodal generative framework for neurodegenerative disease staging. NeurMLLM first encodes the spectrograms and Mel-frequency cepstral coefficients of audio data with vision transformers and projects their representations into the embedding space of a large language model (LLM), where they are concatenated with transcript and demographic instruction tokens as a single unified sequence. The LLM is then instruction-tuned via Low-Rank Adaptation using task prompts to autoregressively predict a constrained label token, enabling a generative classification. By evaluating on the Bridge2AI-Voice dataset for fine-grained staging of AD and PD, we observe that NeurMLLM achieves strong performance, consistently outperforming classical machine learning methods and existing LLM-based approaches. The results show the high potential of multimodal LLMs in neurodegenerative disease staging, improving staging accuracy and supporting accessible deployment.

Related papers

• Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction [23.48690655415839]
  Graph neural networks (GNNs) complement language models (LMs) by incorporating inter-patient relationships and enabling reference-patient attribution.<n>We propose a unified LM--GNN framework that integrates local patient semantics with global cohort structure.<n>Experiments on MIMIC-III and MIMIC-IV show that PLM consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-08-04T19:51:52Z)
• Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification [14.247959730104085]
  multimodal large language models (MLLMs) have sparked an unprecedented wave of applications in the field of medical imaging analysis.<n>However, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models.<n>In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification datasets.
  arXiv  Detail & Related papers  (2026-04-09T15:07:26Z)
• Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models [43.46006663176283]
  This work systematically analyzes the impact of various perturbations on medical MLLMs.<n>For the visual modality, we propose a Perturbation-aware Denoising (PDC) which leverages MLLMs' own vision encoder to identify noise patterns.<n>For text denoising, we design a Self-instantiated Multi-agent System (SMS) that exploits the MLLMs' self-assessment capabilities to refine noisy text.
  arXiv  Detail & Related papers  (2025-12-26T10:23:30Z)
• A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models [85.30893355216486]
  We study how visual token redundancy evolves with different dMLLM architectures and tasks.<n>Our study reveals that visual redundancy emerges only in from-scratch dMLLMs while handling long-answer tasks.<n>Layer-skipping is promising for accelerating AR-to-diffusion dMLLMs, whereas progressive or late-step pruning is more effective for from-scratch dMLLMs.
  arXiv  Detail & Related papers  (2025-11-19T04:13:36Z)
• Multi-Task Learning with Feature-Similarity Laplacian Graphs for Predicting Alzheimer's Disease Progression [13.637114355949356]
  Alzheimer's Disease (AD) is the most prevalent neurodegenerative disorder in aging populations, posing a significant and escalating burden on global healthcare systems.<n>We propose a novel Multi-Tusk Learning (MTL) framework, named Feature Similarity Laplacian graph Multi-Task Learning (MTL-FSL)<n>Our framework introduces a novel Feature Similarity Laplacian (FSL) penalty that explicitly models the time-varying relationships between features.
  arXiv  Detail & Related papers  (2025-10-12T03:55:42Z)
• Leveraging LLMs for Multimodal Retrieval-Augmented Radiology Report Generation via Key Phrase Extraction [0.0]
  We propose a retrieval-augmented generation approach that leverages multimodal retrieval and large language models.<n>Our method uses LLMs to extract key phrases from radiology reports, effectively focusing on essential diagnostic information.<n>We evaluate our approach on MIMIC-CXR dataset, achieving state-of-the-art results on CheXbert metrics and competitive RadGraph F1 metric.
  arXiv  Detail & Related papers  (2025-04-10T03:14:01Z)
• LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition? [59.81732629438753]
  We propose LLaVA-RadZ, a simple yet effective framework for zero-shot medical disease recognition via utilizing the existing MLLM features.<n>Specifically, we design an end-to-end training strategy, termed Decoding-Side Feature Alignment Training (DFAT) to take advantage of the characteristics of the MLLM decoder architecture.<n>We also introduce a Domain Knowledge Anchoring Module (DKAM) to exploit the intrinsic medical knowledge of large models.
  arXiv  Detail & Related papers  (2025-03-10T16:05:40Z)
• Mitigating Hallucinations of Large Language Models in Medical Information Extraction via Contrastive Decoding [92.32881381717594]
  We introduce ALternate Contrastive Decoding (ALCD) to solve hallucination issues in medical information extraction tasks. ALCD demonstrates significant improvements in resolving hallucination issues compared to conventional decoding methods.
  arXiv  Detail & Related papers  (2024-10-21T07:19:19Z)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance [51.30560006045442]
  Image-gRounded guIdaNcE (MARINE) is a framework that is both training-free and API-free.<n>MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs.<n>Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance.
  arXiv  Detail & Related papers  (2024-02-13T18:59:05Z)
• It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition [70.77292069313154]
  Large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF)
  arXiv  Detail & Related papers  (2024-02-08T07:21:45Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.