Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

• URL: http://arxiv.org/abs/2302.05894v2
• Date: Wed, 5 Apr 2023 16:50:50 GMT
• Title: Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia
• Authors: Michail Chatzianastasis, Loukas Ilias, Dimitris Askounis, Michalis Vazirgiannis
• Abstract summary: Leveraging spontaneous speech in conjunction with machine learning methods for recognizing Alzheimer's dementia patients has emerged into a hot topic. Finding a CNN architecture is a time-consuming process and requires expertise. We exploit several fusion methods, including Multimodal Factorized Bilinear Pooling and Tucker Decomposition, to combine both speech and text modalities.
• Score: 14.783829037950984
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Alzheimer's dementia (AD) affects memory, thinking, and language, deteriorating person's life. An early diagnosis is very important as it enables the person to receive medical help and ensure quality of life. Therefore, leveraging spontaneous speech in conjunction with machine learning methods for recognizing AD patients has emerged into a hot topic. Most of the previous works employ Convolutional Neural Networks (CNNs), to process the input signal. However, finding a CNN architecture is a time-consuming process and requires domain expertise. Moreover, the researchers introduce early and late fusion approaches for fusing different modalities or concatenate the representations of the different modalities during training, thus the inter-modal interactions are not captured. To tackle these limitations, first we exploit a Neural Architecture Search (NAS) method to automatically find a high performing CNN architecture. Next, we exploit several fusion methods, including Multimodal Factorized Bilinear Pooling and Tucker Decomposition, to combine both speech and text modalities. To the best of our knowledge, there is no prior work exploiting a NAS approach and these fusion methods in the task of dementia detection from spontaneous speech. We perform extensive experiments on the ADReSS Challenge dataset and show the effectiveness of our approach over state-of-the-art methods.

Related papers

• Diagnosing Alzheimer's Disease using Early-Late Multimodal Data Fusion with Jacobian Maps [1.5501208213584152]
  Alzheimer's disease (AD) is a prevalent and debilitating neurodegenerative disorder impacting a large aging population. We propose an efficient early-late fusion (ELF) approach, which leverages a convolutional neural network for automated feature extraction and random forests. To tackle the challenge of detecting subtle changes in brain volume, we transform images into the Jacobian domain (JD)
  arXiv  Detail & Related papers  (2023-10-25T19:02:57Z)
• Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech [0.0]
  Alzheimer's disease (AD) constitutes a complex neurocognitive disease and is the main cause of dementia. We propose some new methods for detecting AD patients, which capture the intra- and cross-modal interactions. Experiments conducted on the ADReSS and ADReSSo Challenge indicate the efficacy of our introduced approaches over existing research initiatives.
  arXiv  Detail & Related papers  (2023-05-25T18:18:09Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection [69.53626024091076]
  Alzheimer's disease (AD) is particularly prominent in older adults. Recent advances in pre-trained models motivate AD detection modeling to shift from low-level features to high-level representations. This paper presents several efficient methods to extract better AD-related cues from high-level acoustic and linguistic features.
  arXiv  Detail & Related papers  (2023-03-14T16:03:28Z)
• Brain Diffuser: An End-to-End Brain Image to Brain Network Pipeline [54.93591298333767]
  Brain diffuser is a diffusion based end-to-end brain network generative model. It exploits more structural connectivity features and disease-related information by analyzing disparities in structural brain networks across subjects. For the case of Alzheimer's disease, the proposed model performs better than the results from existing toolkits on the Alzheimer's Disease Neuroimaging Initiative database.
  arXiv  Detail & Related papers  (2023-03-11T14:04:58Z)
• A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer [0.0]
  Alzheimer's disease (AD) is a progressive neurological disorder, which affects memory, thinking skills, and mental abilities. We propose deep neural networks, which can be trained in an end-to-end trainable way and capture the inter- and intra-modal interactions.
  arXiv  Detail & Related papers  (2022-11-08T16:43:58Z)
• Cross-Modality Neuroimage Synthesis: A Survey [71.27193056354741]
  Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties. The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research. An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data.
  arXiv  Detail & Related papers  (2022-02-14T19:29:08Z)
• Deep Recurrent Encoder: A scalable end-to-end network to model brain signals [122.1055193683784]
  We propose an end-to-end deep learning architecture trained to predict the brain responses of multiple subjects at once. We successfully test this approach on a large cohort of magnetoencephalography (MEG) recordings acquired during a one-hour reading task.
  arXiv  Detail & Related papers  (2021-03-03T11:39:17Z)
• A Review of Artificial Intelligence Technologies for Early Prediction of Alzheimer's Disease [1.1650381752104297]
  Alzheimer's Disease (AD) is a severe brain disorder, destroying memories and brain functions. The reliable and effective evaluation of early dementia has become essential research with medical imaging technologies and computer-aided algorithms.
  arXiv  Detail & Related papers  (2020-12-22T01:05:34Z)
• Multi-Modal Detection of Alzheimer's Disease from Speech and Text [3.702631194466718]
  We propose a deep learning method that utilizes speech and the corresponding transcript simultaneously to detect Alzheimer's disease (AD) The proposed method achieves 85.3% 10-fold cross-validation accuracy when trained and evaluated on the Dementiabank Pitt corpus.
  arXiv  Detail & Related papers  (2020-11-30T21:18:17Z)
• Spatiotemporal Fusion in 3D CNNs: A Probabilistic View [129.84064609199663]
  We propose to convert success thetemporal fusion strategies into a probability, which allows us to perform network-level evaluations of various fusion strategies without having to train them separately. Our approach greatly boosts the efficiency of analyzingtemporal fusion. We generate new fusion strategies which achieve the state-of-the-art performance on four well-grained action recognition datasets.
  arXiv  Detail & Related papers  (2020-04-10T10:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.