Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention

• URL: http://arxiv.org/abs/2606.09271v1
• Date: Mon, 08 Jun 2026 09:39:33 GMT
• Title: Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention
• Abstract summary: We propose a multi-branch deep learning framework for automatic Parkinson's disease detection from speech.<n>The proposed architecture achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%.
• Score: 9.395262542018235
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria. As speech production relies on the precise coordination of complex neuromuscular mechanisms, speech analysis has emerged as a promising non-invasive and cost-effective biomarker for early PD detection. Recent deep learning approaches have shown encouraging results; however, most existing methods rely on a single speech representation, potentially overlooking complementary pathological information encoded across different feature spaces. In this work, we propose a multi-branch deep learning framework for automatic PD detection from speech. Each recording is segmented into 5-second chunks and represented using three complementary modalities: Log-Mel spectrograms, MFCCs, and HuBERT embeddings extracted from raw waveforms. The spectrograms are processed using a pre-trained ResNet-18 encoder, MFCC sequences are modeled through a BiLSTM network, and raw speech is encoded using a pre-trained HuBERT model. To effectively integrate these heterogeneous representations, we introduce a context-guided cross-modal attention mechanism that dynamically weights temporal HuBERT embeddings according to the global acoustic context derived from the spectrogram and MFCC branches. Experiments conducted on the publicly available Spanish PC-GITA corpus under strict speaker-independent 5-fold cross-validation demonstrate the effectiveness of the proposed approach. The proposed architecture achieves an accuracy of 91.51%, an F1-score of 91.24%, and an AUC of 95.97%. Furthermore, ablation studies confirm the contribution of both the proposed context-guided cross-modal attention mechanism and the integration of complementary speech representations. These findings highlight the potential of heterogeneous speech modeling for robust and clinically reliable PD detection.

Related papers

• Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models [6.567438338350837]
  Speech signals encode both linguistic and acoustic markers associated with cognitive decline.<n>Recent advances in large language models (LLMs) strengthen the potential of speech-based assessment.<n>We propose a multimodal CI detection framework based on open-source LLMs that integrates speech audio and corresponding transcripts.<n>Our work establishes a new state-of-the-art for CI identification, with the proposed method demonstrating superior cross-dataset generalization.
  arXiv  Detail & Related papers  (2026-07-23T16:43:12Z)
• LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features [3.4155322317700585]
  We propose a low-rank adaptation (LoRA)-tuned large language model (LLM) that performs structured multi-view reasoning over four complementary speech-derived signals.<n>On ADReSSo, our best model achieves an F1-score of 90.14%, and ablation confirms the complementary contribution of each view.
  arXiv  Detail & Related papers  (2026-06-26T08:29:31Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• Systematic Study of Dysarthric Speech Recognition: Spectral Features and Acoustic Models [48.10903631277461]
  This paper presents a comprehensive investigation of various combinations of acoustic features tailored to different Acoustic Models.<n>The incorporation of Pitch features notably improved recognition performance, especially for sentence recognition tasks.<n>We have demonstrated the potential to enhance the performance of the state-of-the-art Factorized Time Delay Neural Network (F-TDNN) model for recognizing dysarthric speech.
  arXiv  Detail & Related papers  (2026-06-18T04:57:37Z)
• A Multimodal Framework for Dementia Detection via Linguistic and Acoustic Representation Learning [10.559333552210434]
  Alzheimer's disease is the leading cause of dementia, affecting memory, reasoning, communication, and daily functioning.<n>Recent studies have demonstrated that spontaneous speech contains valuable linguistic and acoustic biomarkers associated with dementia.<n>We propose a multimodal deep learning framework for automatic dementia detection that jointly exploits speech and transcript information in an end-to-end trainable manner.
  arXiv  Detail & Related papers  (2026-05-25T07:57:49Z)
• Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation [76.6507710204181]
  We present Echo-, an agentic multimodal reasoning model for ultrasound interpretation.<n> Echo- is trained to coordinate organ-specific detector outputs, integrate them with global visual context, and convert the resulting evidence into grounded diagnostic decisions.<n>Results suggest that agentic multimodal reasoning can turn specialized detectors into verifiable clinical evidence.
  arXiv  Detail & Related papers  (2026-04-30T15:31:00Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• Explainable Multi-Modal Deep Learning for Automatic Detection of Lung Diseases from Respiratory Audio Signals [0.49581497240446293]
  This study presents an explainable multimodal deep learning framework for automatic lung-disease detection using respiratory audio signals.<n>The framework incorporates Grad-CAM, Integrated Gradients, and SHAP, generating interpretable spectral, temporal, and feature-level explanations.<n>The findings demonstrate the framework's potential for telemedicine, point-of-care diagnostics, and real-world respiratory screening.
  arXiv  Detail & Related papers  (2025-11-29T17:15:58Z)
• From Black Box to Biomarker: Sparse Autoencoders for Interpreting Speech Models of Parkinson's Disease [9.968776083852813]
  Speech holds promise as a cost-effective and non-invasive biomarker for neurological conditions such as Parkinson's disease (PD)<n>We apply sparse autoencoders (SAEs) to uncover interpretable internal representations from a speech-based PD detection system.<n>We introduce a novel mask-based activation for adapting SAEs to small biomedical datasets, creating sparse disentangled dictionary representations.
  arXiv  Detail & Related papers  (2025-07-16T16:22:02Z)
• Detecting Neurocognitive Disorders through Analyses of Topic Evolution and Cross-modal Consistency in Visual-Stimulated Narratives [83.15653194899126]
  Early detection of neurocognitive disorders (NCDs) is crucial for timely intervention and disease management.<n>Current VSN-based NCD detection methods primarily focus on linguistic microstructures closely tied to bottom-up, stimulus-driven cognitive processes.<n>We propose two novel macrostructural approaches: a Dynamic Topic Model (DTM) to track topic evolution over time, and a Text-Image Temporal Alignment Network (TITAN) to measure cross-modal consistency between narrative and visual stimuli.
  arXiv  Detail & Related papers  (2025-01-07T12:16:26Z)
• Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosis [9.91077163490596]
  We propose a novel, interpretable framework specifically designed to support Parkinson's Disease diagnosis.<n>Through the design of simple yet effective cross-attention mechanisms, the proposed framework offers interpretability from two distinct but complementary perspectives.<n>Our method results competitive with state-of-the-art approaches, while also demonstrating robustness in cross-lingual scenarios.
  arXiv  Detail & Related papers  (2024-12-02T22:23:43Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.