Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals

• URL: http://arxiv.org/abs/2411.09243v1
• Date: Thu, 14 Nov 2024 07:20:08 GMT
• Title: Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals
• Authors: Jung-Sun Lee, Ha-Na Jo, Seo-Hyun Lee,
• Abstract summary: This research investigated the effectiveness of deep learning models for non-invasive neural signal decoding. It focused on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech.
• Score: 1.33134751838052
• License:
• Abstract: Brain signals accompany various information relevant to human actions and mental imagery, making them crucial to interpreting and understanding human intentions. Brain-computer interface technology leverages this brain activity to generate external commands for controlling the environment, offering critical advantages to individuals with paralysis or locked-in syndrome. Within the brain-computer interface domain, brain-to-speech research has gained attention, focusing on the direct synthesis of audible speech from brain signals. Most current studies decode speech from brain activity using invasive techniques and emphasize spoken speech data. However, humans express various speech states, and distinguishing these states through non-invasive approaches remains a significant yet challenging task. This research investigated the effectiveness of deep learning models for non-invasive-based neural signal decoding, with an emphasis on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech, across multiple frequency bands. The model utilizing the spatial conventional neural network module demonstrated superior performance compared to other models, especially in the gamma band. Additionally, imagined speech in the theta frequency band, where deep learning also showed strong effects, exhibited statistically significant differences compared to the other speech paradigms.

Related papers

• SIFToM: Robust Spoken Instruction Following through Theory of Mind [51.326266354164716]
  We present a cognitively inspired model, Speech Instruction Following through Theory of Mind (SIFToM), to enable robots to pragmatically follow human instructions under diverse speech conditions. Results show that the SIFToM model outperforms state-of-the-art speech and language models, approaching human-level accuracy on challenging speech instruction following tasks.
  arXiv  Detail & Related papers  (2024-09-17T02:36:10Z)
• Towards Decoding Brain Activity During Passive Listening of Speech [0.0]
  We attempt to decode heard speech from intracranial electroencephalographic (iEEG) data using deep learning methods. This approach diverges from the conventional focus on speech production and instead chooses to investigate neural representations of perceived speech. Despite the approach not having achieved a breakthrough yet, the research sheds light on the potential of decoding neural activity during speech perception.
  arXiv  Detail & Related papers  (2024-02-26T20:04:01Z)
• Neural Speech Embeddings for Speech Synthesis Based on Deep Generative Networks [27.64740032872726]
  We introduce the current brain-to-speech technology with the possibility of speech synthesis from brain signals. Also, we perform comprehensive analysis on the neural features and neural speech embeddings underlying the neurophysiological activation while performing speech.
  arXiv  Detail & Related papers  (2023-12-10T08:12:08Z)
• Decoding speech perception from non-invasive brain recordings [48.46819575538446]
  We introduce a model trained with contrastive-learning to decode self-supervised representations of perceived speech from non-invasive recordings. Our model can identify, from 3 seconds of MEG signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities.
  arXiv  Detail & Related papers  (2022-08-25T10:01:43Z)
• Toward a realistic model of speech processing in the brain with self-supervised learning [67.7130239674153]
  Self-supervised algorithms trained on the raw waveform constitute a promising candidate. We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
  arXiv  Detail & Related papers  (2022-06-03T17:01:46Z)
• Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks [58.24134321728942]
  We compare and identify cognitive aspects on deep neural-based visual lip-reading models. We observe a strong correlation between these theories in cognitive psychology and our unique modeling.
  arXiv  Detail & Related papers  (2021-10-13T05:30:50Z)
• Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects [82.81964713263483]
  A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
  arXiv  Detail & Related papers  (2021-10-12T15:30:21Z)
• Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech [6.87854783185243]
  We compare five types of deep neural networks to human brain responses elicited by spoken sentences. The differences in brain-similarity across networks revealed three main results.
  arXiv  Detail & Related papers  (2021-02-25T19:11:55Z)
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation [57.68765353264689]
  Speech enhancement and speech separation are two related tasks. Traditionally, these tasks have been tackled using signal processing and machine learning techniques. Deep learning has been exploited to achieve strong performance.
  arXiv  Detail & Related papers  (2020-08-21T17:24:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.